AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Last 2048 days  ELOG logo
  ID Date Author Subject
  702   Tue Apr 8 08:52:39 2025 TDAIDA grounding
AIDA DSSSDs and FEE64-DSSSD cabling are electrically isolated (by design) from snout, mounting rods and support assembly.
 FEE64-DSSSD ribbon cables screened by 3M 1245 Cu foil, drain wire to FEE64 adaptor PCB

AIDA FEE64s are electrically isolated (by design) from the AIDA support frame and each other.

BNC PB-5 test signal 
 daisy chained to FEE64 adaptor PCBs via screened RG174 cable/Lemo 00-250 

CAEN N1419ET DSSSD bias
 daisy chained to 1x p+n junction FEE64 adaptor PCB/Si wafer via screened RG174 cable/Lemo 00-250
 CAEN N1419ET outputs floating, locally grounded by 1x n+n Ohmic FEE64 adaptor PCB link/DSSSD

AIDA FEE64s are connected to the DESPEC 19" rack
 power (AIDA FEE64 PSUs),
 timestamp (AIDA NIM MACB modules),
 system console (via serial-USB cables to USB hubs),
 RJ45 network cables (24 port Gbit network switch)

AIDA PSU PowerStax MS1U-6C-222233-01 - see attachments 1-2
 PSU outputs V+/V- and common 
 no direct ground connection

AIDA MACB
 HDMI cabling MACB-FEE64 is screened and shield is grounded

system console
 ?

RJ45 network cables are coupled by ferrite beads to FEE64 PCB and network switch
 no direct ground connection
Attachment 1: powerstax-ms1u_instructionmanua.pdf
powerstax-ms1u_instructionmanua.pdf powerstax-ms1u_instructionmanua.pdf
Attachment 2: 20250408_095624.jpg
20250408_095624.jpg
  701   Tue Mar 11 11:53:18 2025 TDMonday 10 March
18.30 AIDA powered up for brief test

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
       ASIC settings at power up - slow comparator 0x64

      per p+n FEE64 1.8.L spectra - attachment 4
       pulser peak width aida9 55 ch FWHM (39keV FWHm), aida15 53 ch FWHM (37keV FWHM)

      p+n FEE64 aida15 1.8.W spectra - 20us FSR - attachments 5-6

      saved *.L, *.H, *.W spectra for aida15 - attachments 7-11
       *.W spectra - 20us, 200us, 2ms & 20ms FSR 


19.27 DAQ STOP
      DSSSD bias OFF
      FEE64 power OFF
Attachment 1: Screenshot_from_2025-03-10_18-41-32.png
Screenshot_from_2025-03-10_18-41-32.png
Attachment 2: Screenshot_from_2025-03-10_18-42-02.png
Screenshot_from_2025-03-10_18-42-02.png
Attachment 3: Screenshot_from_2025-03-10_18-42-33.png
Screenshot_from_2025-03-10_18-42-33.png
Attachment 4: Screenshot_from_2025-03-10_18-43-07.png
Screenshot_from_2025-03-10_18-43-07.png
Attachment 5: Screenshot_from_2025-03-10_18-48-03.png
Screenshot_from_2025-03-10_18-48-03.png
Attachment 6: Screenshot_from_2025-03-10_18-48-39.png
Screenshot_from_2025-03-10_18-48-39.png
Attachment 7: 2025Mar10-17.59.58.tar.gz
Attachment 8: 2025Mar10-18.05.01.tar.gz
Attachment 9: 2025Mar10-18.06.29.tar.gz
Attachment 10: 2025Mar10-18.12.03.tar.gz
Attachment 11: 2025Mar10-18.17.55.tar.gz
  700   Sat Feb 22 12:35:38 2025 TDOffline analysis data files R9_85-R9_199 (84Mo setting)
Start 00:43 22.2.25 https://elog.gsi.de/despec/G-24-00302/57
AIDA data file R9_85

Converting hexadecimal timestamp to decimal: 1740181266259754800
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Friday, February 21, 2025 11:41:06.259 PM
Your time zone: Friday, February 21, 2025 11:41:06.259 PM GMT+00:00


End 12:56 22.2.25 https://elog.gsi.de/despec/G-24-00302/85 (start of next degrader setting)
AIDA data file R9_199

Converting hexadecimal timestamp to decimal: 1740224934731157800
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Saturday, February 22, 2025 11:48:54.731 AM
Your time zone: Saturday, February 22, 2025 11:48:54.731 AM GMT+00:00

Analysis data files FEB25/R9_85 and FEB25/R9_199
 - no timewarps
 - deadtime aida05 c. 37% and 39% due to high SC41 scaler input to MACB - issue subsequently fixed 
 - deadtime aida04 c. 4%, all other FEE64s <<1%




FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4

n+n Ohmic FEE64s 2, 4

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (507 of 512 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/- 168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   151keV < LEC energy < 1000keV
    to select candidate beta events
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0
   (x,y) strips corresponding to maximum energy
   p+n junction and n+n Ohmic side HEC 

- HEC veto 
   not available - only 1x AIDA DSSSD installed

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 and overall event length < 33us





per FEE64 LEC ADC data items 268ms/channel - attachments 3-5
- all
- 150keV < energy < 1500keV
- energy > 1500keV

absence of high instantaneous rates on spill cf. S181, S100, S505 etc
spill structure visible for energies > 1500keV - probably to be expected as majority of such events should be light ions

per FEE64 HEC ADC data items 268ms/channel - attachments 6-8

- all
- 100MeV < energy < 1000MeV
- energy > 1000MeV

Implant & decay event rates 262us/channel - attachment 9

LEC m_p versus m_n - attachment 10

LEC e_p versus e_n - 20keV/channel - attachment 11

HEC x strip versus y strip - attachment 12

- HEC-LEC implant decay time difference <1s
- HEC-LEC implant decay time difference <100s

shows x-y window used to select 82Nb events

HEC m_p versus m_n - attachment 13

HEC e_p versus e_n - 20MeV/channel - attachment 14


HEC-LEC implant-decay time difference (4ms/channel) versus - attachment 15
- HEC energy 
- LEC energy
- x & y strip 

x strips 130 & 136 disabled

HEC & LEC p strip - n strip time difference (2us/channel, offset=2000 channels) - attachment 16

HEC E (20MeV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 17

LEC E (20keV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 18

LEC e_p - e_n (5.6keV/channel) versus implant-decay time difference (4.194ms/channel) - attachment 19

HEC x & y strip versus implant-decay time difference (4.194ms/channel) - attachment 20

HEC-LEC implant-decay time difference (8.389ms/channel) for candidate 82Nb events - attachment 21
- blue - forward
- cyan - backward

Note
- no FRS PID
- no HEC dE
- no HEC veto
- no HEC energy gates - assume all events within x-y window *stop* in AIDA DSSSD - 82Nb events known to overlap in x-y with (longer-lived, more numerous) 83Nb events
- no front-back time difference
- no clustering
- *all* LEC channels with valid ADC offsets (507 of 512) included *except* 2x strips 130 & 136

HEC-LEC implant-decay time difference (8.389ms/channel) for candidate 82Nb events - attachment 22

Weighted least squares fit channels 1-12 half life 55(9)ms cf. NNDC 50.0(3)ms.
Attachment 1: R9_85.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  143503687 (  365856.8 Hz)
        Other data format:  118416315 (  301897.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2560 (       6.5 Hz)
                               RESUME:       2560 (       6.5 Hz)
                              SYNC100:      33461 (      85.3 Hz)
                              WR48-63:      33461 (      85.3 Hz)
                           FEE64 disc:     326756 (     833.1 Hz)
                             MBS info:  118017517 (  300880.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      29285 (      74.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      392.240 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.069                0.000
  2                0.000                0.000
  3               17.209                0.000
  4              146.817                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.009                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7002448      45530          0          0          0          0        930        930      31901      11769          0       2888
  1   22874309      77583          0          0         18         18       3036       3036      71475          0          0       7512
  2    6498360    4555402          0          0          0          0       1475       1475      34629    4517823          0       3971
  3   82625693      69682          0          0       1306       1306      10819      10819      45432          0          0       3669
  4    4358474  109462549          0          0       1235       1235      14045      14045       2695  109429294          0        206
  5          0          0          0          0          0          0          0          0          0          0          0          0
  6         88          0          0          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0          0          0          0
  8    2927664      19415          0          0          0          0        391        391      18633          0          0       4389
  9   14346055    3938106          0          0          1          1       2392       2392      18452    3914868          0       1270
 10        128          0          0          0          0          0          0          0          0          0          0          0
 11    1956985     169485          0          0          0          0        257        257      25208     143763          0       3888
 12        212          0          0          0          0          0          0          0          0          0          0          0
 13         55          0          0          0          0          0          0          0          0          0          0          0
 14     913148      78563          0          0          0          0        116        116      78331          0          0       1492
 15         68          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   24.117s ( 1326.854 blocks/s,  82.928 Mb/s)
Attachment 2: R9_199.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  154723223 (  435714.6 Hz)
        Other data format:  107196777 (  301875.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2462 (       6.9 Hz)
                               RESUME:       2462 (       6.9 Hz)
                              SYNC100:      33323 (      93.8 Hz)
                              WR48-63:      33323 (      93.8 Hz)
                           FEE64 disc:     288355 (     812.0 Hz)
                             MBS info:  106836852 (  300862.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      32530 (      91.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      355.102 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.008                0.000
  1                0.147                0.000
  2                0.000                0.000
  3               21.597                0.000
  4              136.749                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.006                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9317141      49848          0          0          6          6       1239       1239      36708      10650          0       3256
  1   27128444      89326          0          0         31         31       3596       3596      82072          0          0       8340
  2    7611251    4365052          0          0          0          0       1569       1569      37972    4323942          0       4609
  3   84626002      75259          0          0       1316       1316      10898      10898      50831          0          0       3978
  4    4974948   98704420          0          0       1107       1107      12783      12783       2668   98673972          0        187
  5          0          0          0          0          0          0          0          0          0          0          0          0
  6        122          0          0          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0          0          0          0
  8    3383262      22113          0          0          0          0        436        436      21241          0          0       4811
  9   14397933    3689217          0          0          2          2       2371       2371      21039    3663432          0       1355
 10          1          0          0          0          0          0          0          0          0          0          0          0
 11    2194427     194484          0          0          0          0        298        298      29032     164856          0       4304
 12        140          0          0          0          0          0          0          0          0          0          0          0
 13        260          0          0          0          0          0          0          0          0          0          0          0
 14    1088899       7058          0          0          0          0        133        133       6792          0          0       1690
 15        393          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   26.438s ( 1210.402 blocks/s,  75.650 Mb/s)
Attachment 3: Screenshot_from_2025-03-05_10-04-30.png
Screenshot_from_2025-03-05_10-04-30.png
Attachment 4: Screenshot_from_2025-03-05_10-04-55.png
Screenshot_from_2025-03-05_10-04-55.png
Attachment 5: Screenshot_from_2025-03-05_10-05-29.png
Screenshot_from_2025-03-05_10-05-29.png
Attachment 6: Screenshot_from_2025-03-05_10-07-35.png
Screenshot_from_2025-03-05_10-07-35.png
Attachment 7: Screenshot_from_2025-03-05_10-07-56.png
Screenshot_from_2025-03-05_10-07-56.png
Attachment 8: Screenshot_from_2025-03-05_10-08-29.png
Screenshot_from_2025-03-05_10-08-29.png
Attachment 9: Screenshot_from_2025-03-05_10-09-24.png
Screenshot_from_2025-03-05_10-09-24.png
Attachment 10: Screenshot_from_2025-03-05_10-10-29.png
Screenshot_from_2025-03-05_10-10-29.png
Attachment 11: Screenshot_from_2025-03-05_10-11-17.png
Screenshot_from_2025-03-05_10-11-17.png
Attachment 12: Screenshot_from_2025-03-05_10-12-05.png
Screenshot_from_2025-03-05_10-12-05.png
Attachment 13: Screenshot_from_2025-03-05_10-12-35.png
Screenshot_from_2025-03-05_10-12-35.png
Attachment 14: Screenshot_from_2025-03-05_10-13-38.png
Screenshot_from_2025-03-05_10-13-38.png
Attachment 15: Screenshot_from_2025-03-05_10-14-59.png
Screenshot_from_2025-03-05_10-14-59.png
Attachment 16: Screenshot_from_2025-03-05_10-15-51.png
Screenshot_from_2025-03-05_10-15-51.png
Attachment 17: Screenshot_from_2025-03-08_09-11-40.png
Screenshot_from_2025-03-08_09-11-40.png
Attachment 18: Screenshot_from_2025-03-08_09-11-58.png
Screenshot_from_2025-03-08_09-11-58.png
Attachment 19: Screenshot_from_2025-03-08_09-13-07.png
Screenshot_from_2025-03-08_09-13-07.png
Attachment 20: Screenshot_from_2025-03-08_14-01-50.png
Screenshot_from_2025-03-08_14-01-50.png
Attachment 21: Screenshot_from_2025-03-08_12-21-22.png
Screenshot_from_2025-03-08_12-21-22.png
Attachment 22: 82Nb.png
82Nb.png
  699   Thu Feb 20 17:06:42 2025 JB, CC, TD,MPWR error for aida 13,14,15,16

WR timestamp errors resolved after reseating the HDMI cables to the MACB

  698   Wed Feb 12 10:46:01 2025 TDOffline analysis data files S181 R4_351-396
Analysis data files R4_351-396


Data file R4_351 first WR ts 0x17D7B4C72B0BA9C8
Converting hexadecimal timestamp to decimal: 1718040550378809900
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Monday, June 10, 2024 5:29:10.378 PM
Your time zone: Monday, June 10, 2024 6:29:10.378 PM GMT+01:00 DST
Relative: 8 months ago

Data file R4_396 first WR ts 0x17D7BE86408888E8
Converting hexadecimal timestamp to decimal: 1718051266682718500
Assuming that this timestamp is in nanoseconds (1 billionth of a second):
GMT: Monday, June 10, 2024 8:27:46.682 PM
Your time zone: Monday, June 10, 2024 9:27:46.682 PM GMT+01:00 DST
Relative: 8 months ago

Attachments 1-2 - analysis data files R4_351 and R4_396
 max. *time averaged* deadtime FEE64 #7 (aida08) 1.7% and 1.7% respectively
 all other FEE64 deadtimes < 1%





FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4
DSSSD#2 11  7 16 10 14 13  6  8

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (1012 of 1024 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/- 11200keV (wide open for first pass analysis)

- HEC energy difference +/- 1.68GeV

- valid LEC events

   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1

   LEC energy > 151keV
    to select candidate beta  and alpha events - will include light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 and overall event length < 33us




Number of events observed

 *** scaler # 1 count:  35941696 DSSSD#1 decay events       LEC m_p = 1 and LEC m_n = 1, ADC data > 151.2keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler # 2 count:  31960753 DSSSD#2 decay events       LEC m_p = 1 and LEC m_n = 1, ADC data > 151.2keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler # 3 count:   1499844 DSSSD#1 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler # 4 count:   1297356 DSSSD#2 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler # 5 count:   5446969 DSSSD#1 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler # 6 count:  75030747 DSSSD#2 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8

DSSSD#1 implant events
x=m_p=0 ?
y=m_n=0 ?

DSSSD#2 implant events
x=m_p=0 ?
y=m_n=0 ?


Attachments 3-6 - per FEE64 LEC data item rates 268ms/channel - common x and y scales
 - no conditions
 - 150keV < energy < 1500keV
 - energy > 1500keV

Some deadtime observed on spill, none observed off spill. No significant variation in per FEE64 LEC data rates.

Attachments 7-8 - per FEE64 LEC hit patterns
 - 150keV < energy < 1500keV
 - energy > 1500keV

Aside from usual hot channels at cable/DSSSD boundaries good hit pattern observed. For > 1500keV events observe flat field in x-plane, focussed in y-plane.

Attachments 9-11 - per FEE64 HEC data item rates 268ms/channel - common x and y scales
 - no conditions
 - 100MeV < energy < 1000MeV
 - energy > 1000MeV

Note hot HEC channel in aida08 - should be disabled.

Attachment 12 - per DSSSD decay and implant rates 262us/channel - common x and y scales

Attachment 13 - per DSSSD LEC m_p versus m_n 
 - no conditions

Attachment 14 - per DSSSD LEC p strip versus n strip
 - no conditions

Attachments 15-16 - per DSSSD LEC E_p versus E_n - x and y-axes 20keV/channel
 - LEC energy difference +/- 2000 channels (+/- 11200keV)

Attachments 17-18 - per DSSSD p strip versus n strip
 - HEC-LEC time difference <1s
 - HEC-LEC time difference <100s

Attachment 19 -  per DSSSD HEC m_p versus m_n

Attachment 20 - per DSSSD HEC p strip versus n strip
 - no conditions
 - z_hec=1 => implant stops in DSSSD#1
 - z_hec=3 => implant hits DSSSD#1 and DSSSD#2 but does not necessarily stop in DSSSD#2

Attachment 21 - per DSSSD HEC E_p versus E_n - x and y axes 20MeV/channel

Attachment 22 - DSSSD#1 HEC E_p versus DSSSD#2 HEC E_p - x and y axes 20MeV/channel
 - few events stop in DSSSD#2
 - most events lower Z and A - fission fragments?

Attachment 23 - per DSSSD per pixel HEC-LEC time difference 4.096us/channel
 - events observed to <<100us

Attachment 24 - per DSSSD per pixel HEC-LEC time difference (1s/channel) versus LEC energy (20keV/channel)
 - alpha events long lived 
 - some evidence of effect of on spill deadtime in implant-decay correlations (cf. S100 and S505)

Attachment 25 - per DSSSD implant and decay event p strip - n strip time difference (2us/channel)
 - wider distribution for implant events as expected due to number of ASIC active channels in implant events
 - most decay events +/-2us - lower ASIC occupancy for decay events so most events will be from same (or adjacent) clock cycle
Attachment 1: R4_396.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256709815 ( 1065042.5 Hz)
        Other data format:    5210185 (   21616.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        653 (       2.7 Hz)
                               RESUME:        653 (       2.7 Hz)
                              SYNC100:      32897 (     136.5 Hz)
                              WR48-63:      32897 (     136.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5143085 (   21337.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     639531 (    2653.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      241.032 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.017                0.000
  2                0.000                0.000
  3                0.848                0.000
  4                0.000                0.000
  5                0.227                0.000
  6                0.000                0.000
  7                4.199                0.000
  8                0.000                0.000
  9                1.796                0.000
 10                0.020                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.823                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3315802       8090          0          0          0          0        430        430          0       7230          0       9958
  1    8650533       2148          0          0          3          3       1071       1071          0          0          0      24316
  2    7128165    1944546          0          0          0          0       1082       1082          0    1942382          0      12931
  3   24378546       6508          0          0        152        152       3102       3102          0          0          0      19563
  4    2929004     400994          0          0          0          0        424        424          0     400146          0      12707
  5   19313956       4950          0          0         45         45       2430       2430          0          0          0      22052
  6    5646509     333129          0          0          0          0        723        723          0     331683          0      12637
  7   62853684      16350          0          0        195        195       7980       7980          0          0          0     436912
  8    8105988       2098          0          0          0          0       1049       1049          0          0          0      14497
  9   45373582    1939538          0          0        183        183       6118       6118          0    1926936          0      14193
 10   14519870       3442          0          0         13         13       1708       1708          0          0          0       8558
 11    2400015     535424          0          0          0          0        358        358          0     534708          0      14419
 12    4828613       1128          0          0          0          0        564        564          0          0          0       8079
 13    7222020       1834          0          0          0          0        917        917          0          0          0      12852
 14    3474619        796          0          0          0          0        398        398          0          0          0       8899
 15   36568909       9210          0          0         62         62       4543       4543          0          0          0       6958
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   26.098s ( 1226.164 blocks/s,  76.635 Mb/s)
Attachment 2: R4_351.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256848692 ( 1048248.2 Hz)
        Other data format:    5071308 (   20697.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        624 (       2.5 Hz)
                               RESUME:        624 (       2.5 Hz)
                              SYNC100:      32913 (     134.3 Hz)
                              WR48-63:      32913 (     134.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5004234 (   20423.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     629529 (    2569.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      245.027 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.032                0.000
  2                0.000                0.000
  3                0.744                0.000
  4                0.000                0.000
  5                0.181                0.000
  6                0.000                0.000
  7                4.062                0.000
  8                0.000                0.000
  9                1.739                0.000
 10                0.001                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                1.087                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3260523       8194          0          0          0          0        422        422          0       7350          0       9200
  1    8620748       2224          0          0          6          6       1106       1106          0          0          0      21909
  2    7063739    1926312          0          0          0          0       1116       1116          0    1924080          0      12003
  3   24732934       6660          0          0        134        134       3196       3196          0          0          0      17638
  4    2925688     370692          0          0          0          0        405        405          0     369882          0      11429
  5   18974781       4842          0          0         34         34       2387       2387          0          0          0      20060
  6    5599469     300187          0          0          0          0        782        782          0     298623          0      11800
  7   63229156      16356          0          0        190        190       7988       7988          0          0          0     445010
  8    7941245       2048          0          0          0          0       1024       1024          0          0          0      13314
  9   45626532    1921841          0          0        173        173       5873       5873          0    1909749          0      13056
 10   14656402       3672          0          0          3          3       1833       1833          0          0          0       7698
 11    2320422     495242          0          0          0          0        346        346          0     494550          0      12993
 12    4691764       1142          0          0          0          0        571        571          0          0          0       7316
 13    6904909       1676          0          0          0          0        838        838          0          0          0      11812
 14    3448942        796          0          0          0          0        398        398          0          0          0       7967
 15   36851438       9424          0          0         84         84       4628       4628          0          0          0       6324
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   25.203s ( 1269.684 blocks/s,  79.355 Mb/s)
Attachment 3: Screenshot_from_2025-02-12_08-46-38.png
Screenshot_from_2025-02-12_08-46-38.png
Attachment 4: Screenshot_from_2025-02-12_08-49-32.png
Screenshot_from_2025-02-12_08-49-32.png
Attachment 5: Screenshot_from_2025-02-12_08-50-20.png
Screenshot_from_2025-02-12_08-50-20.png
Attachment 6: Screenshot_from_2025-02-12_08-51-24.png
Screenshot_from_2025-02-12_08-51-24.png
Attachment 7: Screenshot_from_2025-02-12_08-52-57.png
Screenshot_from_2025-02-12_08-52-57.png
Attachment 8: Screenshot_from_2025-02-12_08-53-59.png
Screenshot_from_2025-02-12_08-53-59.png
Attachment 9: Screenshot_from_2025-02-12_08-55-27.png
Screenshot_from_2025-02-12_08-55-27.png
Attachment 10: Screenshot_from_2025-02-12_08-56-03.png
Screenshot_from_2025-02-12_08-56-03.png
Attachment 11: Screenshot_from_2025-02-12_08-56-46.png
Screenshot_from_2025-02-12_08-56-46.png
Attachment 12: Screenshot_from_2025-02-12_08-59-46.png
Screenshot_from_2025-02-12_08-59-46.png
Attachment 13: Screenshot_from_2025-02-12_09-01-08.png
Screenshot_from_2025-02-12_09-01-08.png
Attachment 14: Screenshot_from_2025-02-12_09-01-49.png
Screenshot_from_2025-02-12_09-01-49.png
Attachment 15: Screenshot_from_2025-02-12_09-02-30.png
Screenshot_from_2025-02-12_09-02-30.png
Attachment 16: Screenshot_from_2025-02-12_09-03-06.png
Screenshot_from_2025-02-12_09-03-06.png
Attachment 17: Screenshot_from_2025-02-12_09-04-27.png
Screenshot_from_2025-02-12_09-04-27.png
Attachment 18: Screenshot_from_2025-02-12_09-05-33.png
Screenshot_from_2025-02-12_09-05-33.png
Attachment 19: Screenshot_from_2025-02-12_09-06-04.png
Screenshot_from_2025-02-12_09-06-04.png
Attachment 20: Screenshot_from_2025-02-12_09-06-54.png
Screenshot_from_2025-02-12_09-06-54.png
Attachment 21: Screenshot_from_2025-02-12_09-07-25.png
Screenshot_from_2025-02-12_09-07-25.png
Attachment 22: Screenshot_from_2025-02-12_09-08-02.png
Screenshot_from_2025-02-12_09-08-02.png
Attachment 23: Screenshot_from_2025-02-12_09-08-47.png
Screenshot_from_2025-02-12_09-08-47.png
Attachment 24: Screenshot_from_2025-02-12_09-10-29.png
Screenshot_from_2025-02-12_09-10-29.png
Attachment 25: Screenshot_from_2025-02-12_09-12-14.png
Screenshot_from_2025-02-12_09-12-14.png
  697   Fri Jan 31 17:26:47 2025 CC, TD, MPFriday 31 January contd.
18.18 bPlas and BB7 installs complete 
      Restart AIDA DAQ, Merger, Tape Server and re-test

      per p+n FEE64 1.8.L spectra - attachment 1
       aida09 pulser peak width 69 ch FWHM = 42keV FWHM

      per FEE64 Rate Spectra - attachment 2
       BB18 p+n FEE64s very good/good, n+n FEE64s OK - could be improved
       BB7 aida10 p+n asics good, n+n asics 1x good, 1x OK

      per FEE64 1.8.W spectra - 20us FSR - attachments 3-5

      ADC data item stats - attachment 6
       aida01  aida03 30k, all other BB18 p+n FEE64s < 20k, n+n FEE64s 100/270k
       BB7 aida10 19k

      WR timestamps OK - attachment 7

      Merger, TapeSever etc - attachments 8-9
       disk directpry /TapeData/FEB25
       working OK
     
      DSSSD bias & leakage current OK - attachment 10

      System wide checks OK - attachment 11-15
       note global clock status 6 errors reported earlier today have now gone

      FEE64 temperatures OK - attachment 16

18.38 Transition to safe state

      DAQ STOP
      disable data transfer 1
      detector bias OFF
      FEE64 power OFF


      Can restart as follows

1) FEE64 power ON
2) DAQ RESET
3) DAQ SETUP
4) Enable histogramming
5) Enable waveforms
6) Detector bias ON
7) Restore ASIC settings
8) ASIC Control
9) FEE64 temperatures
10) System wide checks
     sync ASIC clocks
11) FADC control - calibrate ADCS for *all* FEE64s
12) System wide checks contd.
13) DAQ GO
14) Check ADC data item stats
15) Check WR timestamps

If all OK can re-connect to Merger/TapeServer as follows

1) DAQ STOP
2) enable data transfer 1
3) DAQ GO


Can disconnect from Merger/TapeServer as follows

1) DAQ STOP
2) disable data transfer 1
   
Attachment 1: Screenshot_from_2025-01-31_18-18-17.png
Screenshot_from_2025-01-31_18-18-17.png
Attachment 2: Screenshot_from_2025-01-31_18-19-19.png
Screenshot_from_2025-01-31_18-19-19.png
Attachment 3: Screenshot_from_2025-01-31_18-19-37.png
Screenshot_from_2025-01-31_18-19-37.png
Attachment 4: Screenshot_from_2025-01-31_18-20-04.png
Screenshot_from_2025-01-31_18-20-04.png
Attachment 5: Screenshot_from_2025-01-31_18-20-30.png
Screenshot_from_2025-01-31_18-20-30.png
Attachment 6: Screenshot_from_2025-01-31_18-20-42.png
Screenshot_from_2025-01-31_18-20-42.png
Attachment 7: Screenshot_from_2025-01-31_18-21-01.png
Screenshot_from_2025-01-31_18-21-01.png
Attachment 8: Screenshot_from_2025-01-31_18-25-58.png
Screenshot_from_2025-01-31_18-25-58.png
Attachment 9: Screenshot_from_2025-01-31_18-26-13.png
Screenshot_from_2025-01-31_18-26-13.png
Attachment 10: Screenshot_from_2025-01-31_18-29-38.png
Screenshot_from_2025-01-31_18-29-38.png
Attachment 11: Screenshot_from_2025-01-31_18-29-52.png
Screenshot_from_2025-01-31_18-29-52.png
Attachment 12: Screenshot_from_2025-01-31_18-29-58.png
Screenshot_from_2025-01-31_18-29-58.png
Attachment 13: Screenshot_from_2025-01-31_18-30-06.png
Screenshot_from_2025-01-31_18-30-06.png
Attachment 14: Screenshot_from_2025-01-31_18-30-12.png
Screenshot_from_2025-01-31_18-30-12.png
Attachment 15: Screenshot_from_2025-01-31_18-30-30.png
Screenshot_from_2025-01-31_18-30-30.png
Attachment 16: Screenshot_from_2025-01-31_18-30-48.png
Screenshot_from_2025-01-31_18-30-48.png
  696   Fri Jan 31 13:48:29 2025 JBAIDA analysis from HI-DESPEC meeting 20.11.2024

AIDA analysis and presentation by J. Bormans for AIDA made in the HISPEC-DESPEC collaboration meeting.

 

https://docs.google.com/presentation/d/1hlZ30r294UqbVKGy3jg-wompFpc7TSfD/edit?usp=sharing&ouid=102131181856760114019&rtpof=true&sd=true

  695   Fri Jan 31 09:37:47 2025 CC, TD, MPFriday 31 January
Implantation stack mounted - BB18(DS)-1000 + bPlas + bPlas + 3x BB7(DS)-1000

N.B. aida02, aida09 & aida15 have grounded copper screen (3M 1245 - aluminimum braid to copper screen of ribbon cables) for Kapton PCBs connecting the Samtec ribbon cables to the BB18 DSSSD.

n+n FEE64s aida02, aida04, aida06 and aida08 LK1 fitted
p+n FEE64s aida03 and aida07 LK3 fitted

BB7 aida10 (asics #1-2 p+n, #3-4 n+n)

BB18 p+n FEE64s 9-15, 1-3, 5-12 - n+n FEE64s 2-4 (L-R looking downstream) 

10.30 FEE64 power ON

      DSSSD bias ON (BB18 only) - attachment 1
       leakage current OK

      ASIC settings 2024Dec13-17.02.45 restored
       p+n FEE64 slow comparator 0xa, n+n FEE64 slow comparator 0xf

      Attachments 2-7 WR timestamp aida05 & aida12, system wide check
       aida12 & aida05 global clock status 6 - to be checked - OK for initial tests without merger
       WR decoder status aida02

      per FEE64 Rate spectra - attachment 8
       aida10 high rate - BB7 not biased
       rates OK for p+n FEE64s, rates high for n+n FEE64s - to be checked
       BNC PB-5 pulser ON to p+n FEE64s - no obvious adaptor PCB misalignments

      ADC data item stats - attachment 9
       rates generally OK
       aida10 high - BB7 not biased
       aida02 & aida04 n+n FEE64s - rates high - to be checked when bPlas install complete
       all p+n FEE64s connected to BB18 <10K except aida05 25k - very good!

      BNC PB-5 settings - attachment 10

      WR timestamps OK - attachment 11

      FEE64 temperatures OK - attachment 12

12.15 bPlas driver PCB installed, bPlas cabling and grounds *not* connected yet

      ADC data item stats - attachment 13
       all rates higher cf. attachment 9

      per FEE64 Rate spectra - attachment 14

      per p+n FEE64 1.8.L spectra - attachment 15
       aida09 pulser peak width 69 ch FWHM = 49keV FWHM 
       aida14 pulser peak width 50 ch FWHM = 35keV FWHM

      per p+n FEE64 1.8.W spectra - 20us FSR - attachments 16-17

      per n+n FEE64 1.8W spectra - 20us FSR - attachment 18

      DSSSD bias & leakage current OK - attachment 19
       ambient temperature +21.6 deg C, d.p. +0.4 deg C, RH 24.4%

      FEE64 temperatures OK - attachment 20

      DSSSD bias volatge & leakage current OK - attachment 21
       CAEN N1419ET ch#0 BB18, ch#1 BB7

      Install of bPlas drivers, cabling, grounds complete

      ADC data item stats - attachment 22
       
      per FEE64 Rate spectra - attachment 23
       all BB18 p+n FEE64s *except* aida05 show very low rates of noise

      per p+n FEE64 1.8.L spectra - attachment 24
       aida09 pulser peak width 62 ch FWHM = 42keV FWHM
       BB18 p+n FEE64s better electronic noise cf. p+n FEE64s not connected to a DSSSD

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
Attachment 1: Screenshot_from_2025-01-31_10-39-16.png
Screenshot_from_2025-01-31_10-39-16.png
Attachment 2: Screenshot_from_2025-01-31_10-58-48.png
Screenshot_from_2025-01-31_10-58-48.png
Attachment 3: Screenshot_from_2025-01-31_10-58-59.png
Screenshot_from_2025-01-31_10-58-59.png
Attachment 4: Screenshot_from_2025-01-31_10-59-11.png
Screenshot_from_2025-01-31_10-59-11.png
Attachment 5: Screenshot_from_2025-01-31_10-59-11.png
Screenshot_from_2025-01-31_10-59-11.png
Attachment 6: Screenshot_from_2025-01-31_10-59-52.png
Screenshot_from_2025-01-31_10-59-52.png
Attachment 7: Screenshot_from_2025-01-31_11-00-02.png
Screenshot_from_2025-01-31_11-00-02.png
Attachment 8: Screenshot_from_2025-01-31_11-17-59.png
Screenshot_from_2025-01-31_11-17-59.png
Attachment 9: Screenshot_from_2025-01-31_11-18-43.png
Screenshot_from_2025-01-31_11-18-43.png
Attachment 10: Screenshot_from_2025-01-31_11-18-50.png
Screenshot_from_2025-01-31_11-18-50.png
Attachment 11: Screenshot_from_2025-01-31_11-21-05.png
Screenshot_from_2025-01-31_11-21-05.png
Attachment 12: Screenshot_from_2025-01-31_11-21-29.png
Screenshot_from_2025-01-31_11-21-29.png
Attachment 13: Screenshot_from_2025-01-31_12-21-09.png
Screenshot_from_2025-01-31_12-21-09.png
Attachment 14: Screenshot_from_2025-01-31_12-21-55.png
Screenshot_from_2025-01-31_12-21-55.png
Attachment 15: Screenshot_from_2025-01-31_12-26-26.png
Screenshot_from_2025-01-31_12-26-26.png
Attachment 16: Screenshot_from_2025-01-31_12-30-13.png
Screenshot_from_2025-01-31_12-30-13.png
Attachment 17: Screenshot_from_2025-01-31_12-30-34.png
Screenshot_from_2025-01-31_12-30-34.png
Attachment 18: Screenshot_from_2025-01-31_12-31-25.png
Screenshot_from_2025-01-31_12-31-25.png
Attachment 19: Screenshot_from_2025-01-31_12-41-33.png
Screenshot_from_2025-01-31_12-41-33.png
Attachment 20: Screenshot_from_2025-01-31_12-42-08.png
Screenshot_from_2025-01-31_12-42-08.png
Attachment 21: Screenshot_from_2025-01-31_15-59-26.png
Screenshot_from_2025-01-31_15-59-26.png
Attachment 22: Screenshot_from_2025-01-31_16-07-21.png
Screenshot_from_2025-01-31_16-07-21.png
Attachment 23: Screenshot_from_2025-01-31_16-07-31.png
Screenshot_from_2025-01-31_16-07-31.png
Attachment 24: Screenshot_from_2025-01-31_16-08-39.png
Screenshot_from_2025-01-31_16-08-39.png
Attachment 25: Screenshot_from_2025-01-31_16-10-48.png
Screenshot_from_2025-01-31_16-10-48.png
Attachment 26: Screenshot_from_2025-01-31_16-11-30.png
Screenshot_from_2025-01-31_16-11-30.png
  694   Wed Jan 15 11:58:08 2025 TDOffline analysis data files S505 R3_9 - R3_128 (207Hg setting)
Aanlysis data files R3_9 - R3_128 (207Hg setting)

Data file R3_9 first WR ts 0x16FB110A37376D88
GMT: Wednesday, June 22, 2022 10:13:43.562 PM
Your time zone: Wednesday, June 22, 2022 11:13:43.562 PM GMT+01:00 DST

Data file R3_69 first WR ts 0x16FB2286FAD71896
GMT: Thursday, June 23, 2022 3:34:11.118 AM
Your time zone: Thursday, June 23, 2022 4:34:11.118 AM GMT+01:00 DST

Data file R3_128 first WR ts 0x16FB35DF179BD866
GMT: Thursday, June 23, 2022 9:28:40.278 AM
Your time zone: Thursday, June 23, 2022 10:28:40.278 AM GMT+01:00 DST

Attachments 1-3 - analysis data files R3_9, R_69, and R3_128
 max. *time averaged* deadtime FEE64 #1 (aida02) 15.7%, 8.6% and 15.6% respectively
 all other FEE64 deadtimes < 2%


 *** scaler( 1):            120442206  DSSSD#1 decay events       LEC m_p = 1 and LEC m_n = 1, 151.2keV < ADC data < 1008keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler( 2):             51126024  DSSSD#2 decay events       LEC m_p = 1 and LEC m_n = 1, 151.2keV < ADC data < 1008keV *and* HEC m_p = HEC m_n = 0, ADC data > 151.2MeV
 *** scaler( 3):              5630250  DSSSD#1 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler( 4):              4983984  DSSSD#2 implant events     HEC m_p > 0 and HEC m_n > 0, ADC data > 151.2MeV
 *** scaler( 5):             42439339  DSSSD#1 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler( 6):             22316289  DSSSD#2 other events       HEC m_p > 0 or HEC m_n > 0 *and* LEC m_p > 8 or LEC m_n > 8
 *** scaler( 7):                    0
 *** scaler( 8):                    0
 *** scaler( 9):                    0
 *** scaler(10):           2464294601  FEE64 #1 LEC data items
 *** scaler(11):           7641292106
 *** scaler(12):           3547670559
 *** scaler(13):           6144404472
 *** scaler(14):            637262222
 *** scaler(15):           2302716728
 *** scaler(16):           4511293677
 *** scaler(17):           3857002960  FEE64 #8 LEC data items
 *** scaler(18):                    0
 *** scaler(19):                    0
 *** scaler(20):              7697923  FEE64 #1 HEC data items
 *** scaler(21):              7169894
 *** scaler(22):             10028407
 *** scaler(23):              3045622
 *** scaler(24):              4246633
 *** scaler(25):              5564911
 *** scaler(26):              7196158
 *** scaler(27):              1960591  FEE64 #8 HEC data items
 *** scaler(28):                    0
 *** scaler(29):                    0
 *** scaler(30):                    0
 *** scaler(31):                    0
 *** scaler(32):                    0

DSSSD#1 implant events
x=m_p=0 1334587
y=m_n=0 4860983

DSSSD#2 implant events
x=m_p=0 1159736
y=m_n=0 2757735


Attachment 4 - per DSSSD LEC E_p versus E_n - x and y-axes 20keV/channel
 - LEC energy difference +/- 2000 channels (+/- 11200keV)
 - Do not observe the off-axis energy correlations observed in the postrun background/calibration runs
 - Do observe expected inter strip charge sharing and charge sharing at DSSSD active area boundary

All subsequent spectra LEC energy difference +/-20 channels (+/- 112keV)

Attachments 5-7 - per FEE64 LEC data item rates (Hz) 268ms/channel - common x and y scales
 - no conditions
 - 150keV < energy < 1500keV
 - energy > 1500keV
Note structure in > 1500keV data not restricted to on spill

Attachments 8-10 - per FEE64 HEC data item rates (Hz) 268ms/channel - common x and y scales
 - no conditions
 - 100MeV < energy < 1000MeV
 - energy > 1000MeV

Attachment 11 per DSSSD decay and implant rates (Hz) 268ms/channel =- common x and y scales
 - significant on spill deadtime for DSSSD#1

Attachment 12 - per DSSSD LEC m_p versus m_n 
 - no conditions
 - z_hec=1 (DSSSD#1) and z_hec=3 (DSSSD#2)

Attachment 13 - per DSSSD LEC p strip versus n strip

Attachment 14 - per DSSSD LEC E_p versus E_n - x and y axes 20keV/channel

Attachment 15 - per DSSSD per pixel time difference between successive LEC events (4.096us/channel)
 - minimum time difference c. 4us

Attachment 16 -  per DSSSD HEC m_p versus m_n
 - many more m=0 observed cf. RIKEN data e.g. https://elog.ph.ed.ac.uk/AIDA/583
 - more low energy HEC events?

Attachment 17 - per DSSSD HEC p strip versus n strip
 - no conditions
 - z_hec=1 (DSSSD#1) and z_hec=3 (DSSSD#2)
 - z_hec=1 => implant stops in DSSSD#1
 - z_hec=3 => implant hits DSSSD#1 and DSSSD#2 but does not necessarily stop in DSSSD#2

Attachment 18 - per DSSSD HEC E_p versus E_n - x and y axes 20MeV/channel

Attachment 19 - DSSSD#1 HEC E_p versus DSSSD#2 E_p - x and y axes 20MeV/channel

Attachment 20 - DSSSD#1 LEC E_p versus DSSSD#2 E_p - x and y axes 20keV/channel
 - y=0 12816818
 - x=0 7374435
 - x>0 and y>0 2295693
 
Attachment 21 - per DSSSD per pixel time difference between successive HEC events (4.096us/channel)
 - minimum time difference c. 36us which make sense - c. readout time of ASIC with all 16 channels active
 - origin of 9 channel period spectrum structure? 

Attachments 22-23 - per DSSSD per pixel time difference between HEC and LEC events (4.096us/channel)
Attachments 24 - per DSSSD per pixel time difference between HEC and LEC events (1s/channel)
 - for ions stopped in DSSSD#1 or DSSSD'2 - no PID selection
 - minimum time difference c. 88us
 - effect of high on spill deadtime apparent per S100 analysis

Attachment 25 - per DSSSD per pixel time difference between HEC and LEC events (1s/channel) versus
 - HEC energy
 - LEC energy
 - p and n strip

Attachment 26 - per DSSSD implant and decay event p strip - n strip time difference (2us/channel)
 - wider distribution for implant events expected due to number of ASIC active channels in implant events
 - most decay events +/-2us - lower ASIC occupancy for decay events so most events will be from same (or adjacent) clock cycle

Attachment 27 - per DSSSD decay event p strip - n strip time difference (2us/channel) versus decay (E_p - E_n) (20keV channel) 
Attachment 1: R3_9.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      20656
          ADC data format:  167972075 (  847983.1 Hz)
        Other data format:    1097285 (    5539.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        794 (       4.0 Hz)
                               RESUME:        793 (       4.0 Hz)
                              SYNC100:      21394 (     108.0 Hz)
                              WR48-63:      21394 (     108.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1052910 (    5315.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     290405 (    1466.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      198.084 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.300                0.000
  1               31.132                0.000
  2                0.869                0.000
  3                3.536                0.000
  4                0.001                0.000
  5                0.108                0.000
  6                0.157                0.000
  7                0.073                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13716339      17472          0          0         25         25       1820       1820          0      13782          0      46978
  1   39159612      10957          0          0        457        456       5022       5022          0          0          0      45173
  2   20693073     268290          0          0         53         53       2641       2641          0     262902          0      61677
  3   33985341     266701          0          0        182        182       4320       4320          0     257697          0      18991
  4    3620137        906          0          0          1          1        452        452          0          0          0      26130
  5   12686167       3294          0          0         28         28       1619       1619          0          0          0      34215
  6   22242523     257563          0          0         28         28       2791       2791          0     251925          0      44896
  7   21868883     272102          0          0         20         20       2729       2729          0     266604          0      12345
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   15.047s ( 1372.777 blocks/s,  85.799 Mb/s)
Attachment 2: R3_128.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260500382 (  683029.0 Hz)
        Other data format:    1419618 (    3722.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        995 (       2.6 Hz)
                               RESUME:        995 (       2.6 Hz)
                              SYNC100:      33421 (      87.6 Hz)
                              WR48-63:      33421 (      87.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1350786 (    3541.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     304165 (     797.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      381.390 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.726                0.000
  1               32.620                0.000
  2                0.268                0.000
  3                2.352                0.000
  4                0.000                0.000
  5                0.018                0.000
  6                0.191                0.000
  7                0.007                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   27020397      33470          0          0        171        171       3538       3538          0      26052          0      72108
  1   70509491      19838          0          0        649        649       9270       9270          0          0          0      43322
  2   25595248     343140          0          0         48         48       3306       3306          0     336432          0      47964
  3   43087191     344986          0          0         74         74       5493       5493          0     333852          0      26057
  4    4643105       1204          0          0          0          0        602        602          0          0          0      28001
  5   18101121       4580          0          0          9          9       2281       2281          0          0          0      30316
  6   41488735     327203          0          0         39         39       5131       5131          0     316863          0      41967
  7   30055094     345197          0          0          5          5       3800       3800          0     337587          0      14430
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   26.207s ( 1221.046 blocks/s,  76.315 Mb/s)
Attachment 3: R3_69.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260136556 (  817109.0 Hz)
        Other data format:    1783444 (    5601.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1193 (       3.7 Hz)
                               RESUME:       1193 (       3.7 Hz)
                              SYNC100:      33186 (     104.2 Hz)
                              WR48-63:      33186 (     104.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1714686 (    5386.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     465695 (    1462.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      318.362 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.277                0.000
  1               49.537                0.000
  2                0.739                0.000
  3                6.304                0.000
  4                0.000                0.000
  5                0.184                0.000
  6                0.280                0.000
  7                0.034                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   21719834      27795          0          0         49         49       2783       2783          0      22131          0      76464
  1   62193949      17584          0          0        707        707       8085       8085          0          0          0      68411
  2   30630417     438786          0          0         77         77       3925       3925          0     430782          0     101110
  3   52568475     429487          0          0        262        262       6724       6724          0     415515          0      29662
  4    5873809       1474          0          0          0          0        737        737          0          0          0      42098
  5   20399531       5188          0          0         27         27       2567       2567          0          0          0      56158
  6   36994221     420854          0          0         55         55       4590       4590          0     411564          0      72312
  7   29756320     442276          0          0         16         16       3775       3775          0     434694          0      19480
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.504s ( 1361.476 blocks/s,  85.092 Mb/s)
Attachment 4: Screenshot_from_2025-01-14_16-54-08.png
Screenshot_from_2025-01-14_16-54-08.png
Attachment 5: Screenshot_from_2025-01-17_08-18-33.png
Screenshot_from_2025-01-17_08-18-33.png
Attachment 6: Screenshot_from_2025-01-17_08-19-43.png
Screenshot_from_2025-01-17_08-19-43.png
Attachment 7: Screenshot_from_2025-01-17_08-20-32.png
Screenshot_from_2025-01-17_08-20-32.png
Attachment 8: Screenshot_from_2025-01-17_08-21-17.png
Screenshot_from_2025-01-17_08-21-17.png
Attachment 9: Screenshot_from_2025-01-17_08-21-58.png
Screenshot_from_2025-01-17_08-21-58.png
Attachment 10: Screenshot_from_2025-01-17_08-22-35.png
Screenshot_from_2025-01-17_08-22-35.png
Attachment 11: Screenshot_from_2025-01-17_08-23-39.png
Screenshot_from_2025-01-17_08-23-39.png
Attachment 12: Screenshot_from_2025-01-17_08-24-53.png
Screenshot_from_2025-01-17_08-24-53.png
Attachment 13: Screenshot_from_2025-01-17_08-25-25.png
Screenshot_from_2025-01-17_08-25-25.png
Attachment 14: Screenshot_from_2025-01-17_08-25-57.png
Screenshot_from_2025-01-17_08-25-57.png
Attachment 15: Screenshot_from_2025-01-17_08-27-02.png
Screenshot_from_2025-01-17_08-27-02.png
Attachment 16: Screenshot_from_2025-01-17_08-27-34.png
Screenshot_from_2025-01-17_08-27-34.png
Attachment 17: Screenshot_from_2025-01-17_08-28-03.png
Screenshot_from_2025-01-17_08-28-03.png
Attachment 18: Screenshot_from_2025-01-17_08-28-48.png
Screenshot_from_2025-01-17_08-28-48.png
Attachment 19: Screenshot_from_2025-01-17_08-29-18.png
Screenshot_from_2025-01-17_08-29-18.png
Attachment 20: Screenshot_from_2025-01-17_08-29-49.png
Screenshot_from_2025-01-17_08-29-49.png
Attachment 21: Screenshot_from_2025-01-17_08-31-27.png
Screenshot_from_2025-01-17_08-31-27.png
Attachment 22: Screenshot_from_2025-01-17_08-31-57.png
Screenshot_from_2025-01-17_08-31-57.png
Attachment 23: Screenshot_from_2025-01-17_08-32-31.png
Screenshot_from_2025-01-17_08-32-31.png
Attachment 24: Screenshot_from_2025-01-17_08-33-29.png
Screenshot_from_2025-01-17_08-33-29.png
Attachment 25: Screenshot_from_2025-01-17_08-34-12.png
Screenshot_from_2025-01-17_08-34-12.png
Attachment 26: Screenshot_from_2025-01-17_08-35-12.png
Screenshot_from_2025-01-17_08-35-12.png
Attachment 27: Screenshot_from_2025-01-17_08-36-00.png
Screenshot_from_2025-01-17_08-36-00.png
  693   Mon Jan 13 17:38:18 2025 TDOffline analysis of S505 data files
First timestamp of R2_105 0x16FAEC5E54AABF6C

GMT: Wednesday, June 22, 2022 11:01:42.903 AM
Your time zone: Wednesday, June 22, 2022 12:01:42.903 PM GMT+01:00 DST


First timestamp of R3_150 0x16FB3CE72ED53CB6

GMT: Thursday, June 23, 2022 11:37:31.609 AM
Your time zone: Thursday, June 23, 2022 12:37:31.609 PM GMT+01:00 DST


First timestamp of R5_35 0x16FB89F33F9B44E2

GMT: Friday, June 24, 2022 11:09:25.825 AM
Your time zone: Friday, June 24, 2022 12:09:25.825 PM GMT+01:00 DST


First timestamp of R5_299 0x16FBD8CC5DCBE912

GMT: Saturday, June 25, 2022 11:14:20.247 AM
Your time zone: Saturday, June 25, 2022 12:14:20.247 PM GMT+01:00 DST


First timestamp of R5_546 0x16FC220EA8ADB132

GMT: Sunday, June 26, 2022 9:36:49.320 AM
Your time zone: Sunday, June 26, 2022 10:36:49.320 AM GMT+01:00 DST

Attachments 1-5 - analysis of data files R2_105, R3_150, R5_35, R5_299, and R5_536 (c. one data file per day of S505)
 FEE64 #1 (aida02) *time averaged* deadtime c. 12-18%, deadtime of all other FEE64s much lower

Attachments 6-10 - per DSSSD LEC p+n junction versus n+n Ohmic energy - X & y-axes 20keV/channel

No evidence of off-axis energy correlation observed in the postrun background/calibration runs - see https://elog.ph.ed.ac.uk/DESPEC/670
Attachment 1: R2_105.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260590543 (  842959.4 Hz)
        Other data format:    1329457 (    4300.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1451 (       4.7 Hz)
                               RESUME:       1452 (       4.7 Hz)
                              SYNC100:      33151 (     107.2 Hz)
                              WR48-63:      33151 (     107.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1260252 (    4076.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     505847 (    1636.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.138 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                5.956                0.000
  1               46.000                0.000
  2                0.461                0.000
  3                4.075                0.000
  4                0.270                0.000
  5               17.246                0.000
  6                0.939                0.000
  7                0.452                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   27494985      28858          0          0        175        175       3688       3688          0      21132          0     102223
  1   58260423      16474          0          0        649        649       7588       7588          0          0          0      63621
  2   22639873     322607          0          0         55         55       2892       2892          0     316713          0      62482
  3   44542964     319461          0          0        153        154       5539       5539          0     308076          0      35450
  4   10779382       2700          0          0         51         51       1299       1299          0          0          0      87763
  5   28594260       7632          0          0        237        237       3579       3579          0          0          0      64742
  6   38115917     307658          0          0         84         84       4822       4822          0     297846          0      61385
  7   30162739     324067          0          0         47         47       3744       3744          0     316485          0      28181
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   25.383s ( 1260.696 blocks/s,  78.793 Mb/s)
Attachment 2: R3_150.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260385271 (  829605.5 Hz)
        Other data format:    1534729 (    4889.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1425 (       4.5 Hz)
                               RESUME:       1425 (       4.5 Hz)
                              SYNC100:      33170 (     105.7 Hz)
                              WR48-63:      33170 (     105.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1465539 (    4669.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     455452 (    1451.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      313.866 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.666                0.000
  1               57.289                0.000
  2                0.649                0.000
  3                5.611                0.000
  4                0.000                0.000
  5                0.063                0.000
  6                0.228                0.000
  7                0.039                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31441805      29767          0          0        272        272       4062       4062          0      21099          0     112151
  1   66582904      18892          0          0        814        814       8632       8632          0          0          0      66555
  2   29895404     376545          0          0         84         84       3879       3879          0     368619          0      73144
  3   44568333     369126          0          0        168        168       5484       5484          0     357822          0      38819
  4    5095090       1256          0          0          0          0        628        628          0          0          0      37415
  5   19329369       4936          0          0         23         23       2445       2445          0          0          0      43894
  6   36823085     356594          0          0         50         50       4631       4631          0     347232          0      63334
  7   26649281     377613          0          0         14         14       3409       3409          0     370767          0      20140
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.836s ( 1342.511 blocks/s,  83.907 Mb/s)
Attachment 3: R5_35.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260812842 (  842072.4 Hz)
        Other data format:    1107160 (    3574.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1210 (       3.9 Hz)
                               RESUME:       1210 (       3.9 Hz)
                              SYNC100:      33154 (     107.0 Hz)
                              WR48-63:      33154 (     107.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1038432 (    3352.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     329194 (    1062.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.727 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                8.188                0.000
  1               36.773                0.000
  2                0.176                0.000
  3                5.170                0.000
  4                0.000                0.000
  5                0.025                0.000
  6                0.202                0.000
  7                0.307                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31379366      29596          0          0        190        190       4114       4114          0      20988          0      81881
  1   59749297      16628          0          0        605        605       7709       7709          0          0          0      51815
  2   19396332     265009          0          0         42         42       2411       2411          0     260103          0      52064
  3   41019315     263381          0          0        149        149       5195       5195          0     252693          0      27341
  4    4941643       1144          0          0          0          0        572        572          0          0          0      26539
  5   17311877       4276          0          0         10         10       2128       2128          0          0          0      31104
  6   41432200     255213          0          0         55         55       5294       5294          0     244515          0      44402
  7   45582812     271913          0          0        159        159       5731       5731          0     260133          0      14048
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   25.660s ( 1247.070 blocks/s,  77.942 Mb/s)
Attachment 4: R5_299.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260747119 (  773887.2 Hz)
        Other data format:    1172883 (    3481.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1133 (       3.4 Hz)
                               RESUME:       1133 (       3.4 Hz)
                              SYNC100:      33256 (      98.7 Hz)
                              WR48-63:      33256 (      98.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1104105 (    3276.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     362597 (    1076.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      336.932 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                6.724                0.000
  1               44.641                0.000
  2                0.217                0.000
  3                3.561                0.000
  4                0.000                0.000
  5                0.005                0.000
  6                0.223                0.000
  7                0.026                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30674116      31250          0          0        181        181       3927       3927          0      23034          0      91358
  1   66153113      18368          0          0        695        695       8489       8489          0          0          0      55771
  2   19486494     277818          0          0         42         42       2475       2475          0     272784          0      57181
  3   43424107     279556          0          0        135        135       5543       5543          0     268200          0      30417
  4    4766466       1174          0          0          0          0        587        587          0          0          0      29048
  5   17680210       4568          0          0          5          5       2279       2279          0          0          0      34196
  6   42628036     277520          0          0         57         57       5446       5446          0     266514          0      49215
  7   35934577     282629          0          0         18         18       4510       4510          0     273573          0      15411
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   25.258s ( 1266.935 blocks/s,  79.183 Mb/s)
Attachment 5: R5_546.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260730517 (  745151.9 Hz)
        Other data format:    1189483 (    3399.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1144 (       3.3 Hz)
                               RESUME:       1144 (       3.3 Hz)
                              SYNC100:      33304 (      95.2 Hz)
                              WR48-63:      33304 (      95.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1120587 (    3202.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     363968 (    1040.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      349.902 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                6.803                0.000
  1               41.799                0.000
  2                0.301                0.000
  3                3.174                0.000
  4                0.000                0.000
  5                0.022                0.000
  6                0.164                0.000
  7                0.019                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   29065664      31620          0          0        182        182       3682       3682          0      23892          0      90087
  1   70265931      19858          0          0        719        719       9210       9210          0          0          0      57153
  2   19423777     282104          0          0         54         54       2512       2512          0     276972          0      57467
  3   44758741     285792          0          0        125        125       5782       5782          0     273978          0      30202
  4    4551567       1158          0          0          0          0        579        579          0          0          0      29037
  5   17745398       4496          0          0         12         12       2236       2236          0          0          0      34556
  6   43164773     278410          0          0         42         42       5399       5399          0     267528          0      49766
  7   31754666     286045          0          0         10         10       3904       3904          0     278217          0      15700
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   24.879s ( 1286.230 blocks/s,  80.389 Mb/s)
Attachment 6: R2_105.png
R2_105.png
Attachment 7: R3_150.png
R3_150.png
Attachment 8: R5_35.png
R5_35.png
Attachment 9: R5_299.png
R5_299.png
Attachment 10: R5_536.png
R5_536.png
  692   Mon Jan 13 11:32:30 2025 TDOffline analysis S505 data file R3_150
Offline analysis of S505 data files R3_150

first WR ts
First timestamp of R3_150 0x16FB3CE72ED53CB6

Epoch converter says ...

GMT: Thursday, June 23, 2022 11:37:31.609 AM
Your time zone: Thursday, June 23, 2022 12:37:31.609 PM GMT+01:00 DST

last WR ts
First timestamp of R3_151 0x16FB3D3042B9E8E6

Epoch converter says ...

GMT: Thursday, June 23, 2022 11:42:45.475 AM
Your time zone: Thursday, June 23, 2022 12:42:45.475 PM GMT+01:00 DST

Analysis of data file R3_150 - attachment 1
 max time averaged deadtime FEE64 #1 (aida02) 18.3%



FEE64 configuration

FEE64   a  
      b    d
        c

         a  b  c  d 
DSSSD#1  3  4  1  2
DSSSD#2  7  8  5  6

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (474 of 512 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 100keV

- LEC energy difference +/-168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   DSSSD #1
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   DSSSD #2
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1

   151keV < LEC energy < 1008keV
    to select candidate beta events and veto higher energy events e.g. light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500 *or* difference in first and last WR timestamp of event < 33us


Attachment 2 per FEE64 LEC data rate (Hz) 268ms/channel
Attachment 3 per FEE64 LEC data rate (Hz) 268ms/channel: 150keV < energy < 1500keV
Attachment 22 per FEE64 LEC data rate (Hz) 268ms/channel: energy > 1500keV

- observe high instantaneous rate on spill
- rate dominated by low energy (<1500keV) events
- rate of higher energy events dominated by on spill events i.e. light ions as expected
- significant deadtime on spill for n+n Omic FEE64 aida02, low deadtime off spill
- deadtime low/zero for all other FEE64s on/off spill

Attachment 4 per FEE64 HEC data rate (Hz) 268ms/channel
Attachment 5 per FEE64 HEC data rate (Hz) 268ms/channel: 100MeV < energy < 1000MeV
Attachment 6 per FEE64 HEC data rate (Hz) 268ms/channel: energy > 1000MeV

- all HEC events on spill as expected
- significant deadtime on spill for n+n Ohmic FEE64 aida02, low deadtime off spill
- deadtime low/zero for all other FEE64s on/off spill


Attachment 7 per DSSSD decay and implant rate (Hz) 268ms/channel

Attachment 8 log scale
yellow - FEE64 aida01 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida01 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #1 decay rate (Hz) 268ms/channel
green  - DSSSD #1 implant rate (Hz) 268ms/channel

Attachment 9 log scale
yellow - FEE64 aida02 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida02 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #1 decay rate (Hz) 268ms/channel
green  - DSSSD #1 implant rate (Hz) 268ms/channel

Attachment 10 log scale
yellow - FEE64 aida05 LEC data rate (Hz) 268ms/channel
red    - FEE64 aida05 HEC data rate (Hz) 268ms/channel
blue   - DSSSD #2 decay rate (Hz) 268ms/channel
green  - DSSSD #2 implant rate (Hz) 268ms/channel


Attachment 11 per DSSSD LEC m_p versus m_n 

Attachment 12 per DSSSD LEC x versus y 

Attachment 13 per DSSSD LEC p+n junction versus n+n Ohmic energy - x-axis & y-axis 20keV/channel


Attachment 14 per DSSSD HEC m_p versus m_n 

Attachment 15 per DSSSD HEC x versus y 

Attachment 16 per DSSSD HEC p+n junction versus n+n Ohmic energy - x-axis & y-axis 20MeV/channel

Attachment 17 HEC DSSSD#2 p+n junction versus DSSSD#1 p+n junction energy - x-axis & y-axis 20MeV/channel
 to identify which ions stop in DSSSD#2 (z_loc=2)

Attachment 18 decay & implant dx versus dy 

Attachment 19 LEC DSSSD#2 p+n junction versus DSSSD#1 p+n junction energy - x-axis & y-axis 20keV/channel

Attachment 20 per DSSSD decay & implant dt (2us/channel)

Attachment 21 per DSSSD decay dt (2us/channel) versus p+n junctionj - n+n Ohmic energy difference (20keV/channel)


Summary

- high instantaneous data rates on spill
- significant deadtime FEE64 #2 on spill
   magnitude, structure, position of DSSSD#1 decay rate variations differs from FEE64 #2 deadtime - events merging?
- all other FEE64s OK on/off spill
Attachment 1: R3_150.txt
*** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260385271 (  829605.5 Hz)
        Other data format:    1534729 (    4889.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1425 (       4.5 Hz)
                               RESUME:       1425 (       4.5 Hz)
                              SYNC100:      33170 (     105.7 Hz)
                              WR48-63:      33170 (     105.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1465539 (    4669.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     455452 (    1451.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      313.866 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.666                0.000
  1               57.289                0.000
  2                0.649                0.000
  3                5.611                0.000
  4                0.000                0.000
  5                0.063                0.000
  6                0.228                0.000
  7                0.039                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31441805      29767          0          0        272        272       4062       4062          0      21099          0     112151
  1   66582904      18892          0          0        814        814       8632       8632          0          0          0      66555
  2   29895404     376545          0          0         84         84       3879       3879          0     368619          0      73144
  3   44568333     369126          0          0        168        168       5484       5484          0     357822          0      38819
  4    5095090       1256          0          0          0          0        628        628          0          0          0      37415
  5   19329369       4936          0          0         23         23       2445       2445          0          0          0      43894
  6   36823085     356594          0          0         50         50       4631       4631          0     347232          0      63334
  7   26649281     377613          0          0         14         14       3409       3409          0     370767          0      20140
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.836s ( 1342.511 blocks/s,  83.907 Mb/s)
Attachment 2: Screenshot_from_2025-01-12_18-48-06.png
Screenshot_from_2025-01-12_18-48-06.png
Attachment 3: Screenshot_from_2025-01-12_18-49-40.png
Screenshot_from_2025-01-12_18-49-40.png
Attachment 4: Screenshot_from_2025-01-12_18-54-53.png
Screenshot_from_2025-01-12_18-54-53.png
Attachment 5: Screenshot_from_2025-01-12_18-55-44.png
Screenshot_from_2025-01-12_18-55-44.png
Attachment 6: Screenshot_from_2025-01-12_18-56-44.png
Screenshot_from_2025-01-12_18-56-44.png
Attachment 7: Screenshot_from_2025-01-12_18-59-33.png
Screenshot_from_2025-01-12_18-59-33.png
Attachment 8: Screenshot_from_2025-01-12_19-03-44.png
Screenshot_from_2025-01-12_19-03-44.png
Attachment 9: Screenshot_from_2025-01-12_21-25-46.png
Screenshot_from_2025-01-12_21-25-46.png
Attachment 10: Screenshot_from_2025-01-12_19-08-07.png
Screenshot_from_2025-01-12_19-08-07.png
Attachment 11: Screenshot_from_2025-01-12_19-10-12.png
Screenshot_from_2025-01-12_19-10-12.png
Attachment 12: Screenshot_from_2025-01-12_19-10-51.png
Screenshot_from_2025-01-12_19-10-51.png
Attachment 13: Screenshot_from_2025-01-12_19-11-32.png
Screenshot_from_2025-01-12_19-11-32.png
Attachment 14: Screenshot_from_2025-01-12_19-12-14.png
Screenshot_from_2025-01-12_19-12-14.png
Attachment 15: Screenshot_from_2025-01-12_19-12-54.png
Screenshot_from_2025-01-12_19-12-54.png
Attachment 16: Screenshot_from_2025-01-12_19-13-22.png
Screenshot_from_2025-01-12_19-13-22.png
Attachment 17: Screenshot_from_2025-01-12_19-14-02.png
Screenshot_from_2025-01-12_19-14-02.png
Attachment 18: Screenshot_from_2025-01-12_19-14-59.png
Screenshot_from_2025-01-12_19-14-59.png
Attachment 19: Screenshot_from_2025-01-12_19-15-37.png
Screenshot_from_2025-01-12_19-15-37.png
Attachment 20: Screenshot_from_2025-01-12_19-45-39.png
Screenshot_from_2025-01-12_19-45-39.png
Attachment 21: Screenshot_from_2025-01-12_19-46-34.png
Screenshot_from_2025-01-12_19-46-34.png
Attachment 22: Screenshot_from_2025-01-13_20-04-56.png
Screenshot_from_2025-01-13_20-04-56.png
  691   Sat Jan 11 14:14:21 2025 TDaida06 crashed and rebooted 10.1.25
aida06 crashed and rebooted sometime ( 14:48 - 23:25 ) yesterday. System console log appended.


10:01:25/14:58:29|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
10:01:25/14:58:29|------------[ cut here ]------------
10:01:25/23:25:49|kernel BUG at mm/slab.c:2974!
10:01:25/23:25:49|Oops: Exception in kernel mode, sig: 5 [#1]
10:01:25/23:25:50|PREEMPT Xilinx Virtex440
10:01:25/23:25:50|Modules linked in: aidamem xdriver xh_spidev_register
10:01:25/23:25:50|NIP: c009211c LR: c00920b0 CTR: 00000006
10:01:25/23:25:50|REGS: c632dc60 TRAP: 0700   Not tainted  (2.6.31)
10:01:25/23:25:50|MSR: 00021000 <ME,CE>  CR: 22022048  XER: 00000000
10:01:25/23:25:50|TASK = c62064c0[375] 'AidaExecV10' THREAD: c632c000
10:01:25/23:25:50|GPR00: 00000001 c632dd10 c62064c0 c680daf0 c694003c 0000000b c6940020 0000000a 
10:01:25/23:25:50|GPR08: 0000001b c680dae0 00000d80 c680dae0 22008022 10054d4c c03a0000 00000020 
10:01:25/23:25:50|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 
10:01:25/23:25:50|GPR24: 00100100 c632c000 00000000 c680dae8 c680dae0 c680ae00 00000005 c680e400 
10:01:25/23:25:50|NIP [c009211c] cache_alloc_refill+0x130/0x608
10:01:25/23:25:50|LR [c00920b0] cache_alloc_refill+0xc4/0x608
10:01:25/23:25:50|Call Trace:
10:01:25/23:25:50|[c632dd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
10:01:25/23:25:50|[c632dd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
10:01:25/23:25:50|[c632dd90] [c0042420] __sigqueue_alloc+0x50/0xb8
10:01:25/23:25:50|[c632ddb0] [c0042938] __send_signal+0x78/0x260
10:01:25/23:25:50|[c632dde0] [c0042f78] group_send_sig_info+0x70/0x9c
10:01:25/23:25:51|[c632de10] [c00438a8] kill_pid_info+0x48/0x8c
10:01:25/23:25:51|[c632de30] [c0038e8c] it_real_fn+0x1c/0x30
10:01:25/23:25:51|[c632de40] [c0050c40] hrtimer_run_queues+0x184/0x240
10:01:25/23:25:51|[c632dea0] [c0040ba8] run_local_timers+0x10/0x2c
10:01:25/23:25:51|[c632deb0] [c0040bf4] update_process_times+0x30/0x70
10:01:25/23:25:51|[c632ded0] [c005a000] tick_periodic+0x34/0xe8
10:01:25/23:25:51|[c632dee0] [c005a0d4] tick_handle_periodic+0x20/0x120
10:01:25/23:25:51|[c632df20] [c000af70] timer_interrupt+0xa4/0x10c
10:01:25/23:25:51|[c632df40] [c000e9c4] ret_from_except+0x0/0x18
10:01:25/23:25:51|Instruction dump:
10:01:25/23:25:51|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 
10:01:25/23:25:51|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 
10:01:25/23:25:51|Kernel panic - not syncing: Fatal exception in interrupt
10:01:25/23:25:51|Call Trace:
10:01:25/23:25:51|[c632dab0] [c0005de8] show_stack+0x44/0x16c (unreliable)
10:01:25/23:25:51|[c632daf0] [c00345bc] panic+0x94/0x168
10:01:25/23:25:51|[c632db40] [c000bd44] die+0x178/0x18c
10:01:25/23:25:51|[c632db60] [c000c000] _exception+0x164/0x1b4
10:01:25/23:25:51|[c632dc50] [c000e978] ret_from_except_full+0x0/0x4c
10:01:25/23:25:51|[c632dd10] [c00920b0] cache_alloc_refill+0xc4/0x608
10:01:25/23:25:52|[c632dd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
10:01:25/23:25:52|[c632dd90] [c0042420] __sigqueue_alloc+0x50/0xb8
10:01:25/23:25:52|[c632ddb0] [c0042938] __send_signal+0x78/0x260
10:01:25/23:25:52|[c632dde0] [c0042f78] group_send_sig_info+0x70/0x9c
10:01:25/23:25:52|[c632de10] [c00438a8] kill_pid_info+0x48/0x8c
10:01:25/23:25:52|[c632de30] [c0038e8c] it_real_fn+0x1c/0x30
10:01:25/23:25:52|[c632de40] [c0050c40] hrtimer_run_queues+0x184/0x240
10:01:25/23:25:52|[c632dea0] [c0040ba8] run_local_timers+0x10/0x2c
10:01:25/23:25:52|[c632deb0] [c0040bf4] update_process_times+0x30/0x70
10:01:25/23:25:52|[c632ded0] [c005a000] tick_periodic+0x34/0xe8
10:01:25/23:25:52|[c632dee0] [c005a0d4] tick_handle_periodic+0x20/0x120
10:01:25/23:25:52|[c632df20] [c000af70] timer_interrupt+0xa4/0x10c
10:01:25/23:25:52|[c632df40] [c000e9c4] ret_from_except+0x0/0x18
10:01:25/23:25:52|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:28:52|
Finished copying zImage to RAM 
10:01:25/23:28:53|
Found 0 errors checking kernel image
10:01:25/23:28:54|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
10:01:25/23:28:54|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
10:01:25/23:28:54|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
10:01:25/23:28:54|SPIconfig done now to set up the LMK3200 registers
10:01:25/23:28:54|LMK #0 : regInit[0]=0x80000000
10:01:25/23:28:54|LMK #0 : regInit[1]=0x10070600
10:01:25/23:28:54|LMK #0 : regInit[2]=0x60601
10:01:25/23:28:54|LMK #0 : regInit[3]=0x60602
10:01:25/23:28:55|LMK #0 : regInit[4]=0x60603
10:01:25/23:28:55|LMK #0 : regInit[5]=0x70624
10:01:25/23:28:55|LMK #0 : regInit[6]=0x70605
10:01:25/23:28:55|LMK #0 : regInit[7]=0x70606
10:01:25/23:28:55|LMK #0 : regInit[8]=0x70627
10:01:25/23:28:55|LMK #0 : regInit[9]=0x10000908
10:01:25/23:28:55|LMK #0 : regInit[10]=0xA0022A09
10:01:25/23:28:55|LMK #0 : regInit[11]=0x82800B
10:01:25/23:28:55|LMK #0 : regInit[12]=0x28C800D
10:01:25/23:28:55|LMK #0 : regInit[13]=0x830020E
10:01:25/23:28:55|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 942 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
10:01:25/23:28:55|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
10:01:25/23:28:55|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
10:01:25/23:28:55|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
10:01:25/23:28:56|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
10:01:25/23:28:59|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
10:01:25/23:29:05|Using Xilinx Virtex440 machine description
10:01:25/23:29:06|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
10:01:25/23:29:06|Zone PFN ranges:
10:01:25/23:29:06|  DMA      0x00000000 -> 0x00007000
10:01:25/23:29:06|  Normal   0x00007000 -> 0x00007000
10:01:25/23:29:06|Movable zone start PFN for each node
10:01:25/23:29:06|early_node_map[1] active PFN ranges
10:01:25/23:29:06|    0: 0x00000000 -> 0x00007000
10:01:25/23:29:06|MMU: Allocated 1088 bytes of context maps for 255 contexts
10:01:25/23:29:06|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
10:01:25/23:29:06|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
10:01:25/23:29:06|PID hash table entries: 512 (order: 9, 2048 bytes)
10:01:25/23:29:07|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
10:01:25/23:29:07|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
10:01:25/23:29:07|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
10:01:25/23:29:07|Kernel virtual memory layout:
10:01:25/23:29:07|  * 0xffffe000..0xfffff000  : fixmap
10:01:25/23:29:07|  * 0xfde00000..0xfe000000  : consistent mem
10:01:25/23:29:07|  * 0xfde00000..0xfde00000  : early ioremap
10:01:25/23:29:07|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
10:01:25/23:29:07|NR_IRQS:512
10:01:25/23:29:07|clocksource: timebase mult[a00000] shift[22] registered
10:01:25/23:29:07|Console: colour dummy device 80x25
10:01:25/23:29:07|Mount-cache hash table entries: 512
10:01:25/23:29:07|NET: Registered protocol family 16
10:01:25/23:29:07|PCI: Probing PCI hardware
10:01:25/23:29:07|bio: create slab <bio-0> at 0
10:01:25/23:29:07|NET: Registered protocol family 2
10:01:25/23:29:07|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
10:01:25/23:29:07|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
10:01:25/23:29:07|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
10:01:25/23:29:07|TCP: Hash tables configured (established 4096 bind 4096)
10:01:25/23:29:08|TCP reno registered
10:01:25/23:29:08|NET: Registered protocol family 1
10:01:25/23:29:08|ROMFS MTD (C) 2007 Red Hat, Inc.
10:01:25/23:29:08|msgmni has been set to 214
10:01:25/23:29:08|io scheduler noop registered
10:01:25/23:29:08|io scheduler anticipatory registered
10:01:25/23:29:08|io scheduler deadline registered
10:01:25/23:29:08|io scheduler cfq registered (default)
10:01:25/23:29:08|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
10:01:25/23:29:08|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
10:01:25/23:29:08|console [ttyS0] enabled
10:01:25/23:29:08|brd: module loaded
10:01:25/23:29:08|loop: module loaded
10:01:25/23:29:08|Device Tree Probing 'ethernet'
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
10:01:25/23:29:08|XLlTemac: DCR address: 0x80
10:01:25/23:29:08|XLlTemac: buffer descriptor size: 32768 (0x8000)
10:01:25/23:29:08|XLlTemac: Allocating DMA descriptors with kmalloc
10:01:25/23:29:08|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
10:01:25/23:29:08|XTemac: PHY detected at address 7.
10:01:25/23:29:08|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
10:01:25/23:29:09|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09| Intel/Sharp Extended Query Table at 0x010A
10:01:25/23:29:09|Using buffer write method
10:01:25/23:29:09|cfi_cmdset_0001: Erase suspend on write enabled
10:01:25/23:29:09|cmdlinepart partition parsing not available
10:01:25/23:29:09|RedBoot partition parsing not available
10:01:25/23:29:09|Creating 5 MTD partitions on "fc000000.flash":
10:01:25/23:29:09|0x000000000000-0x000000500000 : "golden_firmware"
10:01:25/23:29:09|0x000000500000-0x000000800000 : "golden_kernel"
10:01:25/23:29:09|0x000000800000-0x000000d00000 : "user_firmware"
10:01:25/23:29:09|0x000000d00000-0x000000fe0000 : "user_kernel"
10:01:25/23:29:09|0x000000fe0000-0x000001000000 : "env_variables"
10:01:25/23:29:09|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
10:01:25/23:29:10|SPI: XIlinx spi: bus number now 32766
10:01:25/23:29:10|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
10:01:25/23:29:10|SPI: XIlinx spi: bus number now 32765
10:01:25/23:29:10|mice: PS/2 mouse device common for all mice
10:01:25/23:29:10|Device Tree Probing 'i2c'
10:01:25/23:29:10| #0 at 0x81600000 mapped to 0xD1030000, irq=22
10:01:25/23:29:10|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
10:01:25/23:29:10|TCP cubic registered
10:01:25/23:29:10|NET: Registered protocol family 17
10:01:25/23:29:10|RPC: Registered udp transport module.
10:01:25/23:29:10|RPC: Registered tcp transport module.
10:01:25/23:29:10|eth0: XLlTemac: Options: 0x3fa
10:01:25/23:29:11|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
10:01:25/23:29:11|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
10:01:25/23:29:11|eth0: XLlTemac: speed set to 1000Mb/s
10:01:25/23:29:13|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
10:01:25/23:29:13|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
10:01:25/23:29:13|Sending DHCP requests ., OK
10:01:25/23:29:14|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.6
10:01:25/23:29:14|IP-Config: Complete:
10:01:25/23:29:14|     device=eth0, addr=192.168.11.6, mask=255.255.255.0, gw=255.255.255.255,
10:01:25/23:29:14|     host=aida06, domain=dl.ac.uk, nis-domain=nuclear.physics,
10:01:25/23:29:14|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06
10:01:25/23:29:14|Looking up port of RPC 100003/2 on 192.168.11.99
10:01:25/23:29:14|Looking up port of RPC 100005/1 on 192.168.11.99
10:01:25/23:29:14|VFS: Mounted root (nfs filesystem) on device 0:12.
10:01:25/23:29:14|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
10:01:25/23:29:15|Starting sysinit...
10:01:25/23:29:15|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
10:01:25/23:29:16|		Press 'I' to enter interactive startup.
10:01:25/23:29:16|Setting clock  (utc): Fri Jan 10 22:29:16 GMT 2025 [  OK  ]
10:01:25/23:29:16|Building the cache [  OK  ]
10:01:25/23:29:16|Setting hostname aida06:  [  OK  ]
10:01:25/23:29:18|Mounting local filesystems:  [  OK  ]
10:01:25/23:29:19|Enabling /etc/fstab swaps:  [  OK  ]
10:01:25/23:29:21|Finishing sysinit...
INIT: Entering runlevel: 3
10:01:25/23:29:24|Entering non-interactive startup
10:01:25/23:29:24|FATAL: Module ipv6 not found.
10:01:25/23:29:26|Bringing up loopback interface:  [  OK  ]
10:01:25/23:29:29|FATAL: Module ipv6 not found.
10:01:25/23:29:29|Starting system logger: [  OK  ]
10:01:25/23:29:30|Starting kernel logger: [  OK  ]
10:01:25/23:29:30|Starting rpcbind: [  OK  ]
10:01:25/23:29:32|Mounting NFS filesystems:  [  OK  ]
10:01:25/23:29:32|Mounting other filesystems:  [  OK  ]
10:01:25/23:29:33|Starting xinetd: [  OK  ]
10:01:25/23:29:33|Starting midas:  Starting MIDAS Data Acquisition for aida06
10:01:25/23:29:34|xaida: device parameters: base=0x81000000 size=0x200000
10:01:25/23:29:39|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
10:01:25/23:29:40|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
10:01:25/23:29:40|xaida: driver assigned major number 254
10:01:25/23:29:40|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
10:01:25/23:29:45|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
10:01:25/23:29:45|AIDAMEM: aidamem: driver assigned major number 253
10:01:25/23:29:45|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
10:01:25/23:29:52|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
10:01:25/23:29:52|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
10:01:25/23:29:52|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
10:01:25/23:29:52|Computer Name = aida06; Temp Directory = /tmp/tcl361
10:01:25/23:29:57|
10:01:25/23:30:02|AIDA Data Acquisition Program Release 10.0.Jul  6 2022_15:10:57 starting
10:01:25/23:30:02|
10:01:25/23:30:02|Built without pthreads
10:01:25/23:30:02|
10:01:25/23:30:02|Creating NetVars
10:01:25/23:30:02|Output buffer length = 65504; format option = 4; transfer option = 3
10:01:25/23:30:02|EB transfer option = 3
10:01:25/23:30:02|NetVars created anxaida: open:
10:01:25/23:30:02|d initialised
10:01:25/23:30:02|SAIDAMEM: aidamem_open:
10:01:25/23:30:02|tatistics thread starting
10:01:25/23:30:02|Data Acquisition task has PID 375
10:01:25/23:30:02|Statistics thread created
10:01:25/23:30:02|Stat/Rate creation thread starting
10:01:25/23:30:02|Stat/Rate creation thread created
10:01:25/23:30:02|Hit/Rate creation thread starting
10:01:25/23:30:02|Hit/Rate creation thread created
10:01:25/23:30:02|AIDA Heartbeat thread starting
10:01:25/23:30:02|Heartbeat thread created
10:01:25/23:30:02|Installing signal handlers
10:01:25/23:30:02|Done
10:01:25/23:30:02|ModuleNum = 0
10:01:25/23:30:02|Aida Initialise complete. AidaExecV10.0: Build Jul  6 2022_15:10:57. HDL version : 03350706 
10:01:25/23:30:02|package limit is not available: can't find package limit
10:01:25/23:30:02|Running with default file descriptor limit
10:01:25/23:30:02|Spectra table initialised
10:01:25/23:30:02|AIDA Data Acquisition now all ready to start
10:01:25/23:30:02|SIGBUS, SIGSEGV and SIGPIPE traps setup
10:01:25/23:30:02|package setuid is not available: can't find package setuid
10:01:25/23:30:04|Running as user 0 group 0
10:01:25/23:30:04|[  OK  ]
10:01:25/23:30:12|
10:01:25/23:30:13|DENX ELDK version 4.2 build 2008-04-01
10:01:25/23:30:13|Linux 2.6.31 on a ppc
10:01:25/23:30:13|
10:01:25/23:30:13|aida06 login: /debug user "debug" password "ztl9xf9b3o0l"
10:01:25/23:30:18|httpd started on port 8015
10:01:25/23:30:18|
10:01:25/23:30:18|Cannot use /MIDAS/config/TclHttpd/aida06@8015/startup.tcl
10:01:25/23:30:18|Custom startup from /MIDAS/config/TclHttpd/aida06/startup.tcl
10:01:25/23:30:18|XAIDA Access package 1.0
10:01:25/23:30:20|/XAIDAAccessServer
10:01:25/23:30:21|XAD9252 Access package 1.0
10:01:25/23:30:21|/XAD9252AccessServer
10:01:25/23:30:21|/DataBaseAccessServer
10:01:25/23:30:21|/NetVarService
10:01:25/23:30:21|/SigTaskService
10:01:25/23:30:21|Loaded MemSasAccess
10:01:25/23:30:21|/SpectrumService
10:01:25/23:30:21|loading tcl/AIDARunControl.tcl for namespace ::
10:01:25/23:30:21|/DataAcquisitionControlServer
10:01:25/23:30:21|DefineMessage unknown
10:01:25/23:30:21|Run Control Server Implementation for AIDA
10:01:25/23:30:21|RunControlServer loaded
10:01:25/23:30:21|loading Html/RunControl/implementation.tcl
10:01:25/23:30:22|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
10:01:25/23:30:22|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
10:01:25/23:30:22|Created UI registers
10:01:25/23:30:23|RunControl loaded
10:01:25/23:30:23|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
10:01:25/23:30:23|mkdir returned 1
10:01:25/23:30:23|filepath = /MIDAS/FEE_ASIC
10:01:25/23:30:23|filename = /MIDAS/FEE_ASIC/aida06/ASICs.txt
10:01:25/23:30:23|AIDA RunControl loaded
10:01:25/23:30:24|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
10:01:25/23:30:24|xaida: open:
11:01:25/14:36:47|do_GetState returned z=0 and 8
11:01:25/14:59:08|do_GetState returned z=0 and 8
11:01:25/14:59:20|do_GetState returned z=0 and 8
11:01:25/14:59:32|do_GetState returned z=0 and 8
11:01:25/14:59:44|do_GetState returned z=0 and 8
11:01:25/14:59:56|do_GetState returned z=0 and 8
11:01:25/15:00:08|do_GetState returned z=0 and 8
11:01:25/15:00:20|do_GetState returned z=0 and 8
11:01:25/15:00:33|do_GetState returned z=0 and 8
11:01:25/15:00:45|do_GetState returned z=0 and 8
11:01:25/15:00:57|do_GetState returned z=0 and 8
11:01:25/15:01:09|do_GetState returned z=0 and 8
11:01:25/15:01:21|do_GetState returned z=0 and 8
11:01:25/15:01:33|do_GetState returned z=0 and 8
11:01:25/15:01:45|do_GetState returned z=0 and 8
11:01:25/15:01:58|do_GetState returned z=0 and 8
11:01:25/15:02:10|do_GetState returned z=0 and 8
11:01:25/15:02:22|do_GetState returned z=0 and 8
11:01:25/15:02:34|do_GetState returned z=0 and 8
11:01:25/15:02:46|do_GetState returned z=0 and 8
11:01:25/15:02:59|do_GetState returned z=0 and 8
11:01:25/15:03:11|do_GetState returned z=0 and 8
11:01:25/15:03:23|do_GetState returned z=0 and 8
11:01:25/15:03:35|do_GetState returned z=0 and 8
11:01:25/15:03:47|do_GetState returned z=0 and 8
11:01:25/15:03:59|do_GetState returned z=0 and 8
11:01:25/15:04:11|do_GetState returned z=0 and 8
11:01:25/15:04:24|do_GetState returned z=0 and 8
11:01:25/15:04:36|do_GetState returned z=0 and 8
11:01:25/15:04:48|do_GetState returned z=0 and 8
11:01:25/15:05:00|do_GetState returned z=0 and 8
11:01:25/15:05:12|do_GetState returned z=0 and 8
11:01:25/15:05:24|do_GetState returned z=0 and 8
11:01:25/15:05:36|do_GetState returned z=0 and 8
11:01:25/15:05:49|do_GetState returned z=0 and 8
11:01:25/15:06:01|do_GetState returned z=0 and 8
11:01:25/15:06:13|do_GetState returned z=0 and 8
11:01:25/15:06:25|do_GetState returned z=0 and 8
11:01:25/15:06:37|do_GetState returned z=0 and 8
11:01:25/15:06:50|do_GetState returned z=0 and 8
11:01:25/15:07:02|do_GetState returned z=0 and 8
11:01:25/15:07:14|do_GetState returned z=0 and 8
11:01:25/15:07:26|do_GetState returned z=0 and 8
11:01:25/15:07:38|do_GetState returned z=0 and 8
11:01:25/15:07:51|do_GetState returned z=0 and 8
11:01:25/15:08:03|do_GetState returned z=0 and 8
11:01:25/15:08:15|do_GetState returned z=0 and 8
11:01:25/15:08:27|do_GetState returned z=0 and 8
11:01:25/15:08:40|do_GetState returned z=0 and 8
11:01:25/15:08:52|do_GetState returned z=0 and 8
11:01:25/15:09:04|do_GetState returned z=0 and 8
11:01:25/15:09:17|do_GetState returned z=0 and 8
11:01:25/15:09:29|do_GetState returned z=0 and 8
11:01:25/15:09:41|do_GetState returned z=0 and 8
11:01:25/15:09:53|do_GetState returned z=0 and 8
11:01:25/15:10:06|do_GetState returned z=0 and 8
11:01:25/15:10:18|do_GetState returned z=0 and 8
11:01:25/15:10:30|do_GetState returned z=0 and 8
11:01:25/15:10:43|do_GetState returned z=0 and 8
11:01:25/15:10:55|do_GetState returned z=0 and 8
  690   Sat Jan 11 14:05:03 2025 TDSaturday 11 January 2024
14.37 Detector bias & leakage current OK - attachment 1
  
      FEE64 temperatures OK - attachment 2

      WR timestamps - attachment 3
       aida06 out of sync

      global clock status - attachment 4
       aida06 fails

      ADC calibration - attachment 5
       all fail - probably due to ASIC synchronise *after* ADC calibration

      WR decoder & FPGA TS status - attachments 6-7 
       aida06 status 0xd

      DAQ run control - attachment 8
       aida06 undefined => aida06 has rebooted

      Merger & Tape Server - attachments 9-10
       merger, tape server and data transfer have stopped

      See https://elog.ph.ed.ac.uk/DESPEC/691
       aidao06 system console - aida06 rebooted sometime yesterday


*** Recovery

For FEE64 aida06 *only*

DAQ reset
DAQ setup


For all FEE64s

DAQ stop
Restart merger, merger setup, merger go 
 Following two merger restart cycles all FEE64s stopped OK
Recovery complete

       All system wide checks OK

       Disable data transfer, DAQ go

       per FEE64 rate spectra = attachment 11
        aida04 high rates

       per FEE64 1.8.W spectra - 20us FSR - attachments 12-13

       DAQ stop, enable data transfer, DAQ go

       Merger and server consoles - attachment 14
        merger and tape server (no storage mode) running OK but no data transfer to MBS server

       WR timestamps OK - attachment 15

       ADC and DISC data item stats - attachment 16-17
        aida04 & aida16 high rates - all others appear to be OK
        ASIC check load for all aida16 ASICs changes rate from c. 420k to 0 
        ASIC check load for all aida04 ASICs changes rate from c. 200k to c. 120k

18.00 No data transfer to MBS - nothing useful being donme at the moment - will place system in safe state

      DAQ stop
      Detector bias OFF
      FEE64 power OFF
Attachment 1: Screenshot_from_2025-01-11_14-37-29.png
Screenshot_from_2025-01-11_14-37-29.png
Attachment 2: Screenshot_from_2025-01-11_14-55-21.png
Screenshot_from_2025-01-11_14-55-21.png
Attachment 3: Screenshot_from_2025-01-11_14-56-44.png
Screenshot_from_2025-01-11_14-56-44.png
Attachment 4: Screenshot_from_2025-01-11_14-57-14.png
Screenshot_from_2025-01-11_14-57-14.png
Attachment 5: Screenshot_from_2025-01-11_14-57-37.png
Screenshot_from_2025-01-11_14-57-37.png
Attachment 6: Screenshot_from_2025-01-11_14-58-05.png
Screenshot_from_2025-01-11_14-58-05.png
Attachment 7: Screenshot_from_2025-01-11_14-58-25.png
Screenshot_from_2025-01-11_14-58-25.png
Attachment 8: Screenshot_from_2025-01-11_14-59-24.png
Screenshot_from_2025-01-11_14-59-24.png
Attachment 9: Screenshot_from_2025-01-11_15-00-00.png
Screenshot_from_2025-01-11_15-00-00.png
Attachment 10: Screenshot_from_2025-01-11_15-00-31.png
Screenshot_from_2025-01-11_15-00-31.png
Attachment 11: Screenshot_from_2025-01-11_15-29-32.png
Screenshot_from_2025-01-11_15-29-32.png
Attachment 12: Screenshot_from_2025-01-11_15-30-35.png
Screenshot_from_2025-01-11_15-30-35.png
Attachment 13: Screenshot_from_2025-01-11_15-31-29.png
Screenshot_from_2025-01-11_15-31-29.png
Attachment 14: Screenshot_from_2025-01-11_15-35-14.png
Screenshot_from_2025-01-11_15-35-14.png
Attachment 15: Screenshot_from_2025-01-11_15-38-24.png
Screenshot_from_2025-01-11_15-38-24.png
Attachment 16: Screenshot_from_2025-01-11_15-27-28.png
Screenshot_from_2025-01-11_15-27-28.png
Attachment 17: Screenshot_from_2025-01-11_15-26-39.png
Screenshot_from_2025-01-11_15-26-39.png
  689   Fri Jan 10 10:03:43 2025 TDLEC fast comparator threshold too low
> > > 
> > > LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> > > 
> > > The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> > > 
> > > I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.
> > 
> > Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18
> > 
> > "We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.
> > 
> > This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending 
data. (@Nic is this correct?)
> > 
> > This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."
> 
> We have increased the all thresholds excluding aida10 to 0xff. To not trigger on the other FEEs, however better, the DAQ still occasionally runs into dead time and as such fails with making trees.
> 
> We have thus elected to turn off AIDA and move out the platform. The conclusion is that we are not able to do this fast timing test with a 200 keV threshold.

Per our discussions in December 2024 this is exactly what would be expected.
  688   Thu Jan 9 17:56:14 2025 TDLEC fast comparator threshold too low
> > 
> > LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> > 
> > The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> > 
> > I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.
> 
> Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18
> 
> "We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.
> 
> This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending data. (@Nic is this correct?)
> 
> This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."

We have increased the all thresholds excluding aida10 to 0xff. To not trigger on the other FEEs, however better, the DAQ still occasionally runs into dead time and as such fails with making trees.

We have thus elected to turn off AIDA and move out the platform. The conclusion is that we are not able to do this fast timing test with a 200 keV threshold.
  687   Thu Jan 9 17:36:50 2025 TDLEC fast comparator threshold too low
> 
> LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.
> 
> The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).
> 
> I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.

Thanks. From the DESPEC elog.gsi.de/despec/Implantation+Stack/18

"We have made a trigger from the BGO logic and OR64 of the fast discriminator from AIDA. This was sent as a trigger to the bPlast Exploder.

This did not work as the FEE64s of AIDA are running into high dead-time and a result are dropping data. This is a problem for the ucesb unpacker as it will quite working if one of the subsystem stops sending data. (@Nic is this correct?)

This is indeed what I see as whenever we try to run FAIRROOT the tree builder exits immediately."
  686   Thu Jan 9 16:25:51 2025 TDLEC fast comparator threshold too low
LEC fast comparator has been set to 0x2 (200keV) which is *very* low and means the fast comparator will be triggering on noise.

The *minimum* LEC fast comparator setting should be in the range 0x5 - 0xa (500-1000keV).

I have set the TapeServer to 'no storage' mode to avoid filling the aida-gsi disk with rubbish.
Attachment 1: Screenshot_from_2025-01-09_17-28-45.png
Screenshot_from_2025-01-09_17-28-45.png
  685   Thu Jan 9 15:43:17 2025 JB, CC, MPTiming test platform moved in

16:30 We moved the platform in. The system was still biased and the DAQ was still running.

The system seems to be okay, temperatures OK. Bias voltage OK 685/1 and 685/2. Two FEEs - aida07 and aida12 are receiving a lot of trigger data items from the fast discriminator. This is already a bit worrying. 685/3

Attachment 1: Screenshot_from_2025-01-09_16-41-48.png
Screenshot_from_2025-01-09_16-41-48.png
Attachment 2: Screenshot_from_2025-01-09_16-42-06.png
Screenshot_from_2025-01-09_16-42-06.png
Attachment 3: Screenshot_from_2025-01-09_16-42-46.png
Screenshot_from_2025-01-09_16-42-46.png
  684   Thu Jan 9 12:46:43 2025 JB, MP, CCAIDA timing test

 

Quote:

 

Quote:

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV) 677/11

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

 

 

The pulser was set to 0.5 V to test if we can still see the time spectrum between AIDA and bPlast with reduced thresholds -- mimicking a beta event.

These thresholds were changes from 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4 ---> 0x10, 0x10, 0x0d, 0x10.

This was set to have the HitRate in aida10 to be just above the noise.

 

Initial results: copy from DESPEC elog.

We have managed to obtain the time difference between the bPlast White rabbit and the AIDA Fast time. We had to gate out the zero fast-time events and then also condition that we only take the data from aida10. We see this in 684/1. Then if we look at the time difference between the fast time and the bPlast WRT we see a sharp peak at zero, this makes sense as the AIDA fast time discriminator is being triggered by a pulser which is then being used as a trigger for bPlasts DAQ so these events should be virtually arriving without delay. The delay we do see is infact around 750 ns.

The data collected overnight was also analysed as shown in 684/3 this is the data accumulated with the 22Na source close to the snout and a global threshold in the fast discriminator of 0x0f. The centre peak was roughly fitted with a gaussian:    

  NO.   NAME      VALUE            ERROR          SIZE      DERIVATIVE 
   1  Constant     1.21277e+03   2.74526e+01   5.66915e-01  -5.61074e-07
   2  Mean        -1.78513e+02   8.50699e+00   1.62947e-01   1.58554e-06
   3  Sigma        3.48571e+02   4.03993e+00   6.26343e-05  -1.86863e-03
Attachment 1: AIDAfasttime_bPlastWRdt_AIDA10.png
AIDAfasttime_bPlastWRdt_AIDA10.png
Attachment 2: AIDACalAdcFasttime.png
AIDACalAdcFasttime.png
Attachment 3: AIDAfasttime_bPlastWRdt_AIDA10_nopulser.png
AIDAfasttime_bPlastWRdt_AIDA10_nopulser.png
Attachment 4: Centrepeak_gauss_fit.png
Centrepeak_gauss_fit.png
  683   Wed Jan 8 16:47:05 2025 JB, MP, CCAIDA timing test

 

Quote:

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV) 677/11

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

 

 

The pulser was set to 0.5 V to test if we can still see the time spectrum between AIDA and bPlast with reduced thresholds -- mimicking a beta event.

These thresholds were changes from 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4 ---> 0x10, 0x10, 0x0d, 0x10.

This was set to have the HitRate in aida10 to be just above the noise.

Attachment 1: Screenshot_from_2025-01-08_17-51-51.png
Screenshot_from_2025-01-08_17-51-51.png
  682   Wed Jan 8 10:46:50 2025 TDReport - aida06 -boot fail due to network issue
8:01:25/11:42:48|0x000000d00000-0x000000fe0000 : "user_kernel"
08:01:25/11:42:48|0x000000fe0000-0x000001000000 : "env_variables"
08:01:25/11:42:48|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
08:01:25/11:42:48|SPI: XIlinx spi: bus number now 32766
08:01:25/11:42:48|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
08:01:25/11:42:48|SPI: XIlinx spi: bus number now 32765
08:01:25/11:42:48|mice: PS/2 mouse device common for all mice
08:01:25/11:42:48|Device Tree Probing 'i2c'
08:01:25/11:42:48| #0 at 0x81600000 mapped to 0xD1030000, irq=22
08:01:25/11:42:48|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
08:01:25/11:42:48|TCP cubic registered
08:01:25/11:42:48|NET: Registered protocol family 17
08:01:25/11:42:48|RPC: Registered udp transport module.
08:01:25/11:42:49|RPC: Registered tcp transport module.
08:01:25/11:42:49|eth0: XLlTemac: Options: 0x3fa
08:01:25/11:42:49|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
08:01:25/11:42:49|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
08:01:25/11:42:49|eth0: XLlTemac: speed set to 1000Mb/s
08:01:25/11:42:51|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
08:01:25/11:42:51|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
08:01:25/11:42:51|Sending DHCP requests .
08:01:25/11:42:53|eth0: XLlTemac: PHY Link carrier lost.
08:01:25/11:42:53|..... timed out!
08:01:25/11:44:15|IP-Config: Reopening network devices...
08:01:25/11:44:15|eth0: XLlTemac: Options: 0x3fa
08:01:25/11:44:16|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
08:01:25/11:44:16|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
08:01:25/11:44:16|eth0: XLlTemac: speed set to 1000Mb/s
08:01:25/11:44:18|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
08:01:25/11:44:18|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
08:01:25/11:44:18|Sending DHCP requests ...... timed out!
08:01:25/11:45:35|IP-Config: Auto-configuration of network failed.
08:01:25/11:45:35|Root-NFS: No NFS server available, giving up.
08:01:25/11:45:35|VFS: Unable to mount root fs via NFS, trying floppy.
08:01:25/11:45:35|VFS: Cannot open root device "nfs" or unknown-block(2,0)
08:01:25/11:45:35|Please append a correct "root=" boot option; here are the available partitions:
08:01:25/11:45:35|Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
08:01:25/11:45:35|Call Trace:
08:01:25/11:45:35|[c6827ed0] [c0005de8] show_stack+0x44/0x16c (unreliable)
08:01:25/11:45:35|[c6827f10] [c00345bc] panic+0x94/0x168
08:01:25/11:45:35|[c6827f60] [c0341d34] mount_block_root+0x12c/0x244
08:01:25/11:45:36|[c6827fb0] [c03420d8] prepare_namespace+0x17c/0x208
08:01:25/11:45:36|[c6827fd0] [c0341220] kernel_init+0x104/0x130
08:01:25/11:45:36|[c6827ff0] [c000e140] kernel_thread+0x4c/0x68
08:01:25/11:45:36|Rebooting in 180 seconds..
  681   Wed Jan 8 10:08:29 2025 JB, GB, SD, MP, CC, JGAIDA noise test with platform in position

Yesterday we spent time essentially performing a dry run to get AIDA DAQ and bPlast into the time sorter, there were some issues with the AIDA mbs PC x86l-119 which was related to some boot issue, it was booting to a newer version of debian while the Relay for AIDA to MBS is on an older version (scratch) (the machine is quite old). We spent a bit of time getting the thresholds correct for each of the ASICs. In the end we just elected to have a blanket level of 0x0f for all of the ASICs when we removed the pulser.

Last night we left the setup with the following thresholds on all of the FEEs to collect data. 681/1. The pulser was also turned off and the 22Na source was moved close to the snout.

This morning c. 9:25 we returned and checked the temperatures, HV and statistics and everything seems to be ok. The only problem was that the timesorter has crashed this morning at 8 am.

Attachment 1: Screenshot_from_2025-01-09_09-32-08.png
Screenshot_from_2025-01-09_09-32-08.png
Attachment 2: Screenshot_from_2025-01-09_09-35-11.png
Screenshot_from_2025-01-09_09-35-11.png
Attachment 3: Screenshot_from_2025-01-09_09-32-20.png
Screenshot_from_2025-01-09_09-32-20.png
Attachment 4: Screenshot_from_2025-01-09_09-31-58.png
Screenshot_from_2025-01-09_09-31-58.png
  680   Mon Dec 30 15:43:42 2024 TD[How To] Restart AnyDesk
See also https://elog.ph.ed.ac.uk/CARME/489

1. Establish port (to access carme-gsi)               ssh -L 8080:proxy.gsi.de:8080 carme@atppc025

   Establish port (to access aida-gsi)                ssh -L 3128:proxy.gsi.de:3128 despec@lxlogin

   Once a port is established it will remain accessible (until next system boot?) even if ssh connection drops/closes

2. Check AnyDesk password for remote access has been set up

3. Close anydesk and check it is *really* closed using

ps -o pid= -C anydesk

kill -9 any process (as root)

4 Restart AnyDesk

systemctl restart anydesk (as npg)
  679   Mon Dec 23 10:54:46 2024 TDMonday 23 December 2024
11.54 re-established remote access via AnyDesk - details to follow

      confirmed AIDA FEE64 power OFF, detector HV OFF


1. Establish port (to access carme-gsi)               ssh -L 8080:proxy.gsi.de:8080 carme@atppc025

   Establish port (to access aida-gsi)                ssh -L 3128:proxy.gsi.de:3128 despec@lxlogin

   Once a port is established it will remain accessible (until next system boot?) even if ssh connection drops/closes

2. Check AnyDesk password for remote access has been set up

3. Close anydesk and check it is *really* closed using

ps -o pid= -C anydesk

kill -9 any process (as root)

4 Restart AnyDesk

systemctl restart anydesk (as npg)
  678   Tue Dec 17 12:45:36 2024 JB, MP, CCAIDA timing test

 

Quote:

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV)

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

 

 

678/1 shows the AIDA OR64 trigger from the fast comparator of aida10.

678/2-4 show the bPlast accepted trigger after the bPlast DAQ has been triggered by the AIDA OR64. The images show a signal coming fast in time after the pulser and also a signal at around 80 us that appears to be coming from pile up.

The bPlast DAQ also now runs with the AIDA OR64 trigger after the NIM out signal from the MACB was sent to an octal discriminator to fix the width of the signal and also the pulse width which from TAMEX should be kept at or above 100 ns. The NIM signal from the MACB was mostly 100 ns but also jumping to signals with a 20ns pulse width.

 

Attachment 1: IMG_2536.jpeg
IMG_2536.jpeg
Attachment 2: 20241217_141148.jpg
20241217_141148.jpg
Attachment 3: 20241217_141206.jpg
20241217_141206.jpg
  677   Mon Dec 16 13:07:04 2024 JB, MP, CCAIDA timing test

https://elog.gsi.de/despec/Implantation+Stack/9?suppress=1 - Day 1 ELOG

Day2:

10:14 we set up the detector with the pulser in BB7 and started biasing the detector and setting up the DAQ. Water flow and temperature check, OK. 677/1 677/2 677/3

FEE temps OK. Screenshots included for LOCAL controls for aida07, Discriminator for aida07, ASIC control for aida01 ASIC # 1,2,3,4 and the pulser setting - 1 V @ 10 Hz rep. 677/4 677/5 677/6 677/7 677/8 677/9

ASIC thresholds for the fast comparator LEC/MEC set to 0x20 for all the ASICs in aida10 - connected to BB7. Act on all ASICs did not work with this so each threshold was set to 0x20 on each ASIC by hand.

The pulser is now connected to BB7 test + and via a T-connector to the scope, triggering on the pulser signal. We saw now signals --> lower the threshold of the fast comparator to 1 MeV - 0x64 on all ASICs in aida10. 677/10

No signal was observed, threshold was then set to 0x190 (400 keV) 677/11

13:00 We resumed after lunch. It seems that our thresholds were way too high for starters each channel was actually 100 keV in HEX. We then set the thresholds in aida10 to 0x32, 0x20, 0x20, 0x11 for ASIC 1,2,3,4, respectively. This was to achieve a 10 Hz hit rate in all of the channels! This was done successfully see 677/12 for the hit rate spectrum and also 677/13-16 for the thresholds.

 

 

Attachment 1: Screenshot_from_2024-12-17_10-15-42.png
Screenshot_from_2024-12-17_10-15-42.png
Attachment 2: Screenshot_from_2024-12-17_10-27-00.png
Screenshot_from_2024-12-17_10-27-00.png
Attachment 3: Screenshot_from_2024-12-17_10-21-15.png
Screenshot_from_2024-12-17_10-21-15.png
Attachment 4: Screenshot_from_2024-12-17_10-21-55.png
Screenshot_from_2024-12-17_10-21-55.png
Attachment 5: Screenshot_from_2024-12-17_10-22-20.png
Screenshot_from_2024-12-17_10-22-20.png
Attachment 6: Screenshot_from_2024-12-17_10-23-27.png
Screenshot_from_2024-12-17_10-23-27.png
Attachment 7: Screenshot_from_2024-12-17_10-23-32.png
Screenshot_from_2024-12-17_10-23-32.png
Attachment 8: Screenshot_from_2024-12-17_10-23-37.png
Screenshot_from_2024-12-17_10-23-37.png
Attachment 9: Screenshot_from_2024-12-17_10-23-42.png
Screenshot_from_2024-12-17_10-23-42.png
Attachment 10: Screenshot_from_2024-12-17_10-35-07.png
Screenshot_from_2024-12-17_10-35-07.png
Attachment 11: Screenshot_from_2024-12-17_10-39-36.png
Screenshot_from_2024-12-17_10-39-36.png
Attachment 12: Screenshot_from_2024-12-17_13-35-40.png
Screenshot_from_2024-12-17_13-35-40.png
Attachment 13: Screenshot_from_2024-12-17_13-35-45.png
Screenshot_from_2024-12-17_13-35-45.png
Attachment 14: Screenshot_from_2024-12-17_13-35-51.png
Screenshot_from_2024-12-17_13-35-51.png
Attachment 15: Screenshot_from_2024-12-17_13-35-56.png
Screenshot_from_2024-12-17_13-35-56.png
Attachment 16: Screenshot_from_2024-12-17_13-36-01.png
Screenshot_from_2024-12-17_13-36-01.png
  676   Sat Dec 14 11:52:42 2024 TDSaturday 14 December
11:00 Visual inspection FEE64 adaptor PCBs & cabling
      aida02 ground from/to other FEE64s disconnected - re-connected
      aida01, aida09, aida12 - ground cabling screws to Lemo 00.250 housings loose - tightemed

12:50 Cooling water pressure & temperature OK - attachments 1  2

12:59 relay #1 power ON

13:01 relay #2 power ON

13:07 aida06 starts - panic during startup, automatic restart following 3 minute timeout

      DAQ reset, setup

      Check ASIC Control - browser tab timeout 
      AIDA MIDAS HTTPD server console log - attachment 3

      Appears to have restored ASIC settings 2024Dec13-17.02.45 saved yesterday
    
      aida10 ASICs #1 & #2 positive input, ASICs #3 & #4 negative input
      slow comparator 0xa (all p+n junction FEE64s and aida10), 0xf (n+n Ohmic FEE64s)

13:28 tar ASIC settings - attachment 4

[npg@aidas-gsi]$ cd /MIDAS/FEE_ASIC
[npg@aidas-gsi FEE_ASIC]$ tar cvf /tmp/FEE_ASIC.tar .

       System wide checks

       Sync ASIC clocks - attachment 5

       Clock, ADC calibration, WR decoder, FPGA timestamp, PLL checks - attachments 6-11
        all OK *except* aida02 WR decoder error

       WR timestamps OK - attachment 12
       FEE64 temps OK - attachment 13
       
13:45 Detector bias ON - attachment 14      

      BNC PB-5 pulser settings - attachment 15
      Pulser connected to all p+n junction FEE64s *except* aida10

      ADC, DISC, PAUSE and MBS correlation scaler stats - attachments 16-19
       aida02 rate significantly lower than yesterday - https://elog.gsi.de/despec/Implantation+Stack/8
       high rates observed for aida08, aida11 and aida14 - which are not connected to a DSSSD!

      per FEE64 Rate spectra - attachment 20

      per p+n junction FEE64 1.8.L spectra - attachment 21
       aida09 pulser peak width 56 ch FWHM = 39 keV FWHM
       consistent electronic noise for all p+n junction FEE64s (cabling+DSSSD)
       electronic noise of p+n junction FEE64s (cabling *only*) higher and more variable cf. https://elog.gsi.de/despec/Implantation+Stack/8 attachment 5

      per p+n junction FEE64 1.8.W spectra 20us FSR - attachments 22-23

      per n+n Ohmic FEE64 1.8.W spectra 20us FSR - attachment 24 

      WR timestamps OK - attachment 25

14:43 DAQ STOP
      Data transfer enabled
      Select Tape Server -> Next Run
      DAQ GO  data file R3

      Merger, Tape Server - attachments 26-27
       data transfer rate c. 900k data items/s cf c. 300k data items/s yesterday https://elog.gsi.de/despec/Implantation+Stack/8

16:45 DAQ STOP

      Data transfer disabled 

      Detector bias OFF

      FEE64 power OFF
Attachment 1: 20241214_125048.jpg
20241214_125048.jpg
Attachment 2: 20241214_125104.jpg
20241214_125104.jpg
Attachment 3: asic_check.log
executing do_PreReset
 ::MASTERTS namespace already exists
done executing do_PreReset
executing do_PostSetup
done executing do_PostSetup 
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida01
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida01
Second load re-activated at 0x0000401c  for FEE aida01 
Shifting out finished at 0 counts 
Read the control register from FEE aida01 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida01
Second load re-activated at 0x0000405c  for FEE aida01 
Shifting out finished at 0 counts 
Read the control register from FEE aida01 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida01
Second load re-activated at 0x0000409c  for FEE aida01 
Shifting out finished at 0 counts 
Read the control register from FEE aida01 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida01
Second load re-activated at 0x000040dc  for FEE aida01 
Shifting out finished at 0 counts 
Read the control register from FEE aida01 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida02
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida02
Second load re-activated at 0x0000401c  for FEE aida02 
Shifting out finished at 0 counts 
Read the control register from FEE aida02 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida02
Second load re-activated at 0x0000405c  for FEE aida02 
Shifting out finished at 0 counts 
Read the control register from FEE aida02 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida02
Second load re-activated at 0x0000409c  for FEE aida02 
Shifting out finished at 0 counts 
Read the control register from FEE aida02 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida02
Second load re-activated at 0x000040dc  for FEE aida02 
Shifting out finished at 0 counts 
Read the control register from FEE aida02 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida03
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida03
Second load re-activated at 0x0000401c  for FEE aida03 
Shifting out finished at 0 counts 
Read the control register from FEE aida03 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida03
Second load re-activated at 0x0000405c  for FEE aida03 
Shifting out finished at 0 counts 
Read the control register from FEE aida03 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida03
Second load re-activated at 0x0000409c  for FEE aida03 
Shifting out finished at 0 counts 
Read the control register from FEE aida03 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida03
Second load re-activated at 0x000040dc  for FEE aida03 
Shifting out finished at 0 counts 
Read the control register from FEE aida03 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida04
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida04
Second load re-activated at 0x0000401c  for FEE aida04 
Shifting out finished at 0 counts 
Read the control register from FEE aida04 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida04
Second load re-activated at 0x0000405c  for FEE aida04 
Shifting out finished at 0 counts 
Read the control register from FEE aida04 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida04
Second load re-activated at 0x0000409c  for FEE aida04 
Shifting out finished at 0 counts 
Read the control register from FEE aida04 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida04
Second load re-activated at 0x000040dc  for FEE aida04 
Shifting out finished at 0 counts 
Read the control register from FEE aida04 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida05
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida05
Second load re-activated at 0x0000401c  for FEE aida05 
Shifting out finished at 0 counts 
Read the control register from FEE aida05 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida05
Second load re-activated at 0x0000405c  for FEE aida05 
Shifting out finished at 0 counts 
Read the control register from FEE aida05 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida05
Second load re-activated at 0x0000409c  for FEE aida05 
Shifting out finished at 0 counts 
Read the control register from FEE aida05 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida05
Second load re-activated at 0x000040dc  for FEE aida05 
Shifting out finished at 0 counts 
Read the control register from FEE aida05 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida06
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida06
Second load re-activated at 0x0000401c  for FEE aida06 
Shifting out finished at 0 counts 
Read the control register from FEE aida06 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida06
Second load re-activated at 0x0000405c  for FEE aida06 
Shifting out finished at 0 counts 
Read the control register from FEE aida06 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida06
Second load re-activated at 0x0000409c  for FEE aida06 
Shifting out finished at 0 counts 
Read the control register from FEE aida06 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida06
Second load re-activated at 0x000040dc  for FEE aida06 
Shifting out finished at 0 counts 
Read the control register from FEE aida06 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida07
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida07
Second load re-activated at 0x0000401c  for FEE aida07 
Shifting out finished at 0 counts 
Read the control register from FEE aida07 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida07
Second load re-activated at 0x0000405c  for FEE aida07 
Shifting out finished at 0 counts 
Read the control register from FEE aida07 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida07
Second load re-activated at 0x0000409c  for FEE aida07 
Shifting out finished at 0 counts 
Read the control register from FEE aida07 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida07
Second load re-activated at 0x000040dc  for FEE aida07 
Shifting out finished at 0 counts 
Read the control register from FEE aida07 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida08
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida08
Second load re-activated at 0x0000401c  for FEE aida08 
Shifting out finished at 0 counts 
Read the control register from FEE aida08 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida08
Second load re-activated at 0x0000405c  for FEE aida08 
Shifting out finished at 0 counts 
Read the control register from FEE aida08 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida08
Second load re-activated at 0x0000409c  for FEE aida08 
Shifting out finished at 0 counts 
Read the control register from FEE aida08 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida08
Second load re-activated at 0x000040dc  for FEE aida08 
Shifting out finished at 0 counts 
Read the control register from FEE aida08 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida09
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida09
Second load re-activated at 0x0000401c  for FEE aida09 
Shifting out finished at 0 counts 
Read the control register from FEE aida09 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida09
Second load re-activated at 0x0000405c  for FEE aida09 
Shifting out finished at 0 counts 
Read the control register from FEE aida09 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida09
Second load re-activated at 0x0000409c  for FEE aida09 
Shifting out finished at 0 counts 
Read the control register from FEE aida09 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida09
Second load re-activated at 0x000040dc  for FEE aida09 
Shifting out finished at 0 counts 
Read the control register from FEE aida09 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida10
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida10
Second load re-activated at 0x0000401c  for FEE aida10 
Shifting out finished at 0 counts 
Read the control register from FEE aida10 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida10
Second load re-activated at 0x0000405c  for FEE aida10 
Shifting out finished at 0 counts 
Read the control register from FEE aida10 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida10
Second load re-activated at 0x0000409c  for FEE aida10 
Shifting out finished at 0 counts 
Read the control register from FEE aida10 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida10
Second load re-activated at 0x000040dc  for FEE aida10 
Shifting out finished at 0 counts 
Read the control register from FEE aida10 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida11
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida11
Second load re-activated at 0x0000401c  for FEE aida11 
Shifting out finished at 0 counts 
Read the control register from FEE aida11 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida11
Second load re-activated at 0x0000405c  for FEE aida11 
Shifting out finished at 0 counts 
Read the control register from FEE aida11 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida11
Second load re-activated at 0x0000409c  for FEE aida11 
Shifting out finished at 0 counts 
Read the control register from FEE aida11 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida11
Second load re-activated at 0x000040dc  for FEE aida11 
Shifting out finished at 0 counts 
Read the control register from FEE aida11 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida12
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida12
Second load re-activated at 0x0000401c  for FEE aida12 
Shifting out finished at 0 counts 
Read the control register from FEE aida12 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida12
Second load re-activated at 0x0000405c  for FEE aida12 
Shifting out finished at 0 counts 
Read the control register from FEE aida12 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida12
Second load re-activated at 0x0000409c  for FEE aida12 
Shifting out finished at 0 counts 
Read the control register from FEE aida12 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida12
Second load re-activated at 0x000040dc  for FEE aida12 
Shifting out finished at 0 counts 
Read the control register from FEE aida12 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida13
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida13
Second load re-activated at 0x0000401c  for FEE aida13 
Shifting out finished at 0 counts 
Read the control register from FEE aida13 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida13
Second load re-activated at 0x0000405c  for FEE aida13 
Shifting out finished at 0 counts 
Read the control register from FEE aida13 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida13
Second load re-activated at 0x0000409c  for FEE aida13 
Shifting out finished at 0 counts 
Read the control register from FEE aida13 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida13
Second load re-activated at 0x000040dc  for FEE aida13 
Shifting out finished at 0 counts 
Read the control register from FEE aida13 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida14
checking asic 1
Loaded ASIC 1 at 0x0000401c for FEE aida14
Second load re-activated at 0x0000401c  for FEE aida14 
Shifting out finished at 0 counts 
Read the control register from FEE aida14 ASIC 1
checking asic 2
Loaded ASIC 2 at 0x0000405c for FEE aida14
Second load re-activated at 0x0000405c  for FEE aida14 
Shifting out finished at 0 counts 
Read the control register from FEE aida14 ASIC 2
checking asic 3
Loaded ASIC 3 at 0x0000409c for FEE aida14
Second load re-activated at 0x0000409c  for FEE aida14 
Shifting out finished at 0 counts 
Read the control register from FEE aida14 ASIC 3
checking asic 4
Loaded ASIC 4 at 0x000040dc for FEE aida14
Second load re-activated at 0x000040dc  for FEE aida14 
Shifting out finished at 0 counts 
Read the control register from FEE aida14 ASIC 4
ASIC.tcl :- CheckASIC fetched file /MIDAS/FEE_ASIC => aida15
... 379 more lines ...
Attachment 4: FEE_ASIC.tar
Attachment 5: Screenshot_from_2024-12-14_13-30-48.png
Screenshot_from_2024-12-14_13-30-48.png
Attachment 6: Screenshot_from_2024-12-14_13-30-54.png
Screenshot_from_2024-12-14_13-30-54.png
Attachment 7: Screenshot_from_2024-12-14_13-31-02.png
Screenshot_from_2024-12-14_13-31-02.png
Attachment 8: Screenshot_from_2024-12-14_13-35-22.png
Screenshot_from_2024-12-14_13-35-22.png
Attachment 9: Screenshot_from_2024-12-14_13-35-28.png
Screenshot_from_2024-12-14_13-35-28.png
Attachment 10: Screenshot_from_2024-12-14_13-35-33.png
Screenshot_from_2024-12-14_13-35-33.png
Attachment 11: Screenshot_from_2024-12-14_13-36-06.png
Screenshot_from_2024-12-14_13-36-06.png
Attachment 12: Screenshot_from_2024-12-14_13-36-27.png
Screenshot_from_2024-12-14_13-36-27.png
Attachment 13: Screenshot_from_2024-12-14_13-36-42.png
Screenshot_from_2024-12-14_13-36-42.png
Attachment 14: Screenshot_from_2024-12-14_13-44-29.png
Screenshot_from_2024-12-14_13-44-29.png
Attachment 15: Screenshot_from_2024-12-14_13-46-41.png
Screenshot_from_2024-12-14_13-46-41.png
Attachment 16: Screenshot_from_2024-12-14_13-54-58.png
Screenshot_from_2024-12-14_13-54-58.png
Attachment 17: Screenshot_from_2024-12-14_13-55-19.png
Screenshot_from_2024-12-14_13-55-19.png
Attachment 18: Screenshot_from_2024-12-14_13-55-33.png
Screenshot_from_2024-12-14_13-55-33.png
Attachment 19: Screenshot_from_2024-12-14_13-56-00.png
Screenshot_from_2024-12-14_13-56-00.png
Attachment 20: Screenshot_from_2024-12-14_14-10-08.png
Screenshot_from_2024-12-14_14-10-08.png
Attachment 21: Screenshot_from_2024-12-14_14-14-18.png
Screenshot_from_2024-12-14_14-14-18.png
Attachment 22: Screenshot_from_2024-12-14_14-26-30.png
Screenshot_from_2024-12-14_14-26-30.png
Attachment 23: Screenshot_from_2024-12-14_14-27-16.png
Screenshot_from_2024-12-14_14-27-16.png
Attachment 24: Screenshot_from_2024-12-14_14-29-24.png
Screenshot_from_2024-12-14_14-29-24.png
Attachment 25: Screenshot_from_2024-12-14_14-42-40.png
Screenshot_from_2024-12-14_14-42-40.png
Attachment 26: Screenshot_from_2024-12-14_14-43-51.png
Screenshot_from_2024-12-14_14-43-51.png
Attachment 27: Screenshot_from_2024-12-14_14-43-58.png
Screenshot_from_2024-12-14_14-43-58.png
  675   Wed Dec 11 14:46:21 2024 TD, JB, CC, MPMounting of AIDA, bPlast and BB7 for test

Snout assembled for detector position and timing test.

The planned test is to use a BGO with a 22-Na source and the implantation stack to characterise the timing and position of back-to-back 511 keV events (proxy for mock beta decay event).

The implantation stack is assembled as follows:

Upstream ----> AIDA test frame (as a spacer) ----> AIDA DSSSD (3208-3/3208-21/3208-22, Upstream AIDA used in S100 & S181, April 2024) ----> bPlast (Downstream bPlast) ----> BB7 (mounted in test frame connected to AIDA adaptor) ----> Downstream

The AIDA DSSSD condition seems to be the same as when it was dismounted after S181 and it, apart from some specks of dust no bond wires appear to be broken to the naked eye.

BGO signals appear on the scope with a 22-Na source but the response is quite messy.

 

 

17:36

MSL type BB18(DS)-1000 24cm  x 8cm DSSSD upstream position within snout using 8x upstream (23") ribbon cables 

MSL type BB(DS)-1000 positioned downstream of downstream bPlast within snout using 1x downstream (29") ribbon cable. For BB7 'isolated' Right Coupler Kapton PCB replaced by unmodified Right Coupler Kapton PCB.

 

 

 

Attachment 1: 20241211_161837.jpg
20241211_161837.jpg
Attachment 2: 20241211_161833.jpg
20241211_161833.jpg
Attachment 3: 20241211_161830.jpg
20241211_161830.jpg
Attachment 4: 20241211_161820.jpg
20241211_161820.jpg
  674   Mon Dec 9 12:28:01 2024 NH, CC, MPTest of BB7 on AIDA Electronics
BB7 connected to aida07, sitting in a cardboard box

Update Tcl/Tml files in ASIC, ASIC4 and sys.tml frm PJCS email to allow setting per-ASIC polarity on FEE64
Restart MIDAS@8015

Power on all FEEs

Note nnrpi is not responding to ssh... MP investigates if it ahs been unplugged
 No, but a powercycle worked and fixed it

See error: /MIDAS/FEE_ASIC/aida06/ASICs.txt  not found, create the folders

We see some data in the L spectra, and changing the settings from one ASIC seems to work properly

Stopped for CC/MP to determine the which ASICs correspond to p+n and n+n

Follow up: Alpha data overnight or source?
  673   Thu Dec 5 11:18:46 2024 TDOffline analysis S100 data files R21_0-R21_99
Offline analysis of S100 data files R21_0 - R21_99 (162Eu setting)

first WR ts
First timestamp of R21_0 0x17CA09154AE3E636

Epoch converter says ...

GMT: Saturday, April 27, 2024 4:36:35.223 AM
Your time zone: Saturday, April 27, 2024 5:36:35.223 AM GMT+01:00 DST

last WR ts
First timestamp of R21_100 0x17CA16C1904150CE

GMT: Saturday, April 27, 2024 8:47:08.772 AM
Your time zone: Saturday, April 27, 2024 9:47:08.772 AM GMT+01:00 DST


FEE64 configuration

FEE64   a b c 
      g       h
        d e f

         a  b  c  d  e  f  g  h
DSSSD#1 15  3 12  9  1  5  2  4
DSSSD#2 11  7 16 10 14 13  6  8

n+n Ohmic FEE64s 2, 4, 6, 8

Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (1012 of 1024 channels)
      LEC calibration ADC offset only

- no clustering

- no multiplex timestamp correction

- no p+n junction side - n+n Ohmic side correlation time gates

- FEE64 *not* DSSSD strip ordering

- hardware - slow comparator setting p+n junction FEE64s 100keV, n+n Ohmic FEE64s 150keV

- LEC energy difference +/-168keV

- HEC energy difference +/- 1.68GeV

- valid LEC events

   DSSSD #1
   p+n junction side multiplicity = 1 and n+n Ohmic side multiplicity = 1
   DSSSD #2 
   0 < p+n junction side multiplicity < 8 
   and
   0 < n+n Ohmic side multiplicity < 8 

   151keV < LEC energy < 1008keV
    to select candidate beta events and veto higher energy events e.g. light ions
    standalone analysis of AIDA data, no downstream veto detector

- valid HEC events
   p+n junction side multiplicity > 0 and n+n Ohmic side multiplicity > 0

  (x,y) strips corresponding to maximum energy
  p+n junction and n+n Ohmic side HEC 

- HEC veto 
   p+n junction side multiplicity > 0 or n+n Ohmic side multiplicity > 0

- per pixel implant-decay correlations

- end of event 
   difference in WR timestamp between successive ADC data items > 2500

Attachments 1-4
per DSSSD p+n junction - n+n Ohmic strip time difference for HEC and LEC events (2us/channel) linear and log scale

- observe large (> 32us) time differences (on log scale)

- range of time differences increases with multiplicity ( DSSSD#1 cf. DSSSD#2 LEC events)

- distribution of HEC time differences can probably be understood in terms of most/all channels of ASIC being active during HEC event with low LEC thresholds

- AIDA is a triggerless DAQ producing streams of ADC data items *not* events
   at high instantaneous rates when events are constructed they may become aggregated in time i.e. > 32us readout time of all channels of one ASIC

- To investigate impose additional end of event criterion 
   difference in first and last WR timestamp of event < 33us

Attachments 5-6
per DSSSD p+n junction - n+n Ohmic strip time difference for HEC and LEC events (2us/channel) linear and log scale

- blue original end of event criteria, cyan new end of event criteria

- as expected range of time differences is restricted to +/- 32us 

- observe somewhat higher fraction of events with low time differences
 
  DSSSD #1 10363098 of 16104322 (64%) events +/-2us 

  DSSSD #2 860454912 of 1766618199 (49%) events +/-2us

Attachment 7
per DSSSD p+n junction - n+n Ohmic strip time difference for LEC events - x-axis 2us/channel, y-axis 20keV /channel


Attachment 8 per FEE64 LEC data rate (Hz) 268ms/channel
Attachment 9 per FEE64 LEC data rate (Hz) 268ms/channel: 150keV < energy < 1500keV
Attachment 10 per FEE64 LEC data rate (Hz) 268ms/channel: energy > 1500keV

- observe high instantaneous rate on spill
- rate dominated by low energy (<1500keV) events
- rate of higher energy events dominated by on spill events i.e. light ions as expected
- significant deadtime on spill for n+n Ohmic FEE64s, low deadtime off spill
- deadtime low/zero for p+n junction FEE64s on/off spill

Attachment 11 per FEE64 LEC hit pattern: 150keV < energy < 1500keV
Attachment 12 per FEE64 LEC hit pattern: energy > 1500keV


Attachment 13 per FEE64 HEC data rate (Hz) 268ms/channel
Attachment 14 per FEE64 HEC data rate (Hz) 268ms/channel: 100MeV < energy < 1000MeV
Attachment 15 per FEE64 HEC data rate (Hz) 268ms/channel: energy > 1000MeV

- rate dominated by low energy (>1GeV) events
- all HEC events on spill as expected (note FEE64 #7 has a single hot channel which can be disabled in software)
- significant deadtime on spill for n+n Ohmic FEE64s, low deadtime off spill
- deadtime low/zero for p+n junction FEE64s on/off spill

Attachment 16 per DSSSD p+n junction versus n+n Ohmic LEC energy - x-axis & y-axis 20keV/channel

Attachment 17 per DSSSD p+n junction versus n+n Ohmic HEC energy - x-axis & y-axis 20MeV/channel

Attachment 18 per DSSSD p+n junction versus n+n Ohmic HEC strip hit pattern: all HEC events

Attachment 19 per DSSSD p+n junction versus n+n Ohmic HEC strip hit pattern
 DSSSD #1 ions stopped in DSSSD #1 i.e. DSSSD #2 HEC multiplicity = 0
 DSSSD #1 shows x-y gate used ( 270 < x < 370, 20 < y < 90 ) to identify 166Tb implants
 DSSSD #2 ions stopped in DSSSD #2 *and* in transmission (can establish which ions stop in DSSSD#2 from DSSSD#2 HEC energy versus DSSSD#1 HEC energy - see https://elog.ph.ed.ac.uk/DESPEC/672

Attachment 20

DSSSD#1 HEC energy (20MeV/channel) versus HEC-LEC dt (1s/channel)

DSSSD#1 LEC energy (20keV/channel) versus HEC-LEC dt (1s/channel)

DSSSD#1 HEC strip # versus HEC-LEC dt (1s/channel)

- Observe # events in every third channel is lower 
- Probably reflects implant-decay correlation livetime
  For example (choosing some numbers for illustrative purposes)
  on spill: HEC livetime 75%, LEC livetime 75% (FEE64 deadtime common for HEC and LEC data) => implant decay correlation livetime 56%
  off spill: HEC live time 75%, LEC livetime 100% =>  implant decay correlation livetime 75%
- Observe 'hot' x channels 315, 318, 321, 324 - disabled for further analysis
- Do not observe any 'hot' y channels 


Attachments 21 & 22

DSSSD#1 per pixel HEC-LEC time (1s/channel): x,y,z gated to select 166Tb events

Naive (parent-daughter decay only, flat background) fit for data t=0-26s ( t1/2 = 27.1(3)s )

Fit ignores data for t=0, 3, 6, 10, 13, 16, 19, 22s to avoid bias of differences in implant-decay correlation deadtime

Suggestion of structure at c. 30s period? Does this reflect spill stucture? 10x spill cycles (30s), 9s spill off, ... etc

Sum of x,y,z gated HEC events (s2112 - see attachment 19) = 670441

Elapsed time of dataset 4h11m = 15060s
# pixels = 100 x 70 = 7000
=> # x,y,z gated HEC events per pixel = 670441/7000/15060 = 0.0064/s or mean time between x,y,z gated HEC event = 157s (estimate needs to be corrected for HEC deadtime)

Sum of implant decay correlations (s2220 - see attachment 21) t=0-150s = 273508 - flat background estimated as 150 x 500 = 75000 = 198508

=> efficiency c. 30% (presumably low due to implant-decay deadtime, LEC multiplicity, per pixel correlations and no clustering)


Summary

$64,000 question - what is the origin of the high instantaneous rate on spill  for DSSSD#1 ? On my to do list.


Attachment 23

LEC multiplicity with/without HEC data in event

per DSSSD LEC p+n junction multiplicity versus n+n Ohmic multiplicity
per DSSSD LEC p+n junction multiplicity versus n+n Ohmic multiplicity z_hec=1 and z_hec=2


With HEC data

DSSSD#1 p+n junction multiplicity ~ 17, n+n Ohmic multiplicity ~28
DSSSD#2 p+n junction multiplicity ~ 40, n+n Ohmic multiplicity ~23

Assume 200Hz HEC events => DSSSD#1 LEC rate = 200 x ( 17 + 28 ) => 9k LEC data items cf. >100k LEC data items (attachments 8 & 13)

i.e. not due to HEC events








 
 


 







   
Attachment 1: Screenshot_from_2024-12-05_11-17-50.png
Screenshot_from_2024-12-05_11-17-50.png
Attachment 2: Screenshot_from_2024-12-05_11-18-13.png
Screenshot_from_2024-12-05_11-18-13.png
Attachment 3: Screenshot_from_2024-12-05_11-04-40.png
Screenshot_from_2024-12-05_11-04-40.png
Attachment 4: Screenshot_from_2024-12-05_11-05-03.png
Screenshot_from_2024-12-05_11-05-03.png
Attachment 5: Screenshot_from_2024-12-05_11-07-58.png
Screenshot_from_2024-12-05_11-07-58.png
Attachment 6: Screenshot_from_2024-12-05_11-09-55.png
Screenshot_from_2024-12-05_11-09-55.png
Attachment 7: Screenshot_from_2024-12-04_16-56-02.png
Screenshot_from_2024-12-04_16-56-02.png
Attachment 8: Screenshot_from_2024-12-04_16-39-09.png
Screenshot_from_2024-12-04_16-39-09.png
Attachment 9: Screenshot_from_2024-12-04_16-39-59.png
Screenshot_from_2024-12-04_16-39-59.png
Attachment 10: Screenshot_from_2024-12-04_16-41-20.png
Screenshot_from_2024-12-04_16-41-20.png
Attachment 11: Screenshot_from_2024-12-04_16-41-54.png
Screenshot_from_2024-12-04_16-41-54.png
Attachment 12: Screenshot_from_2024-12-04_16-42-37.png
Screenshot_from_2024-12-04_16-42-37.png
Attachment 13: Screenshot_from_2024-12-04_16-43-36.png
Screenshot_from_2024-12-04_16-43-36.png
Attachment 14: Screenshot_from_2024-12-04_16-44-13.png
Screenshot_from_2024-12-04_16-44-13.png
Attachment 15: Screenshot_from_2024-12-04_16-44-50.png
Screenshot_from_2024-12-04_16-44-50.png
Attachment 16: Screenshot_from_2024-12-04_16-46-26.png
Screenshot_from_2024-12-04_16-46-26.png
Attachment 17: Screenshot_from_2024-12-04_16-50-32.png
Screenshot_from_2024-12-04_16-50-32.png
Attachment 18: Screenshot_from_2024-12-04_16-49-12.png
Screenshot_from_2024-12-04_16-49-12.png
Attachment 19: Screenshot_from_2024-12-04_16-49-57.png
Screenshot_from_2024-12-04_16-49-57.png
Attachment 20: Screenshot_from_2024-12-04_16-52-09.png
Screenshot_from_2024-12-04_16-52-09.png
Attachment 21: Screenshot_from_2024-12-04_16-51-30.png
Screenshot_from_2024-12-04_16-51-30.png
Attachment 22: 166Tb.png
166Tb.png
Attachment 23: Screenshot_from_2024-12-07_16-57-28.png
Screenshot_from_2024-12-07_16-57-28.png
  672   Wed Dec 4 09:57:24 2024 TDHISPEC DESPEC meeting presentation - November 2024 -0 Summary of AIDA performance 2024
Attachment 1: Summary_of_AIDA_2024_distro.pdf
Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf Summary_of_AIDA_2024_distro.pdf
  671   Tue Nov 19 11:24:25 2024 TDAnydesk restarted remotely
Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489
  670   Tue Nov 5 15:19:35 2024 TDS505 offline analysis R5_780 - R5_814
Offline analysis of data files S505/R5_780 - R5_814 (corresponding to MBS data files 73-74) 

Can find an analysis of alpha background runs at RIBF, RIKEN for comparison at https://elog.ph.ed.ac.uk/AIDA/816

Beam off - background runs without sources

MBS 73 27.6.22 09:33-11:33 CEST https://elog.gsi.de/despec/S505/204

MBS 74 27.6.22 11:34-13:34 CEST https://elog.gsi.de/despec/S505/206

MBS 75 27.6.22 13:35-16:29 CEST https://elog.gsi.de/despec/S505/209

ADC offsets per https://elog.ph.ed.ac.uk/DESPEC/556

FEE64 configuration

FEE64   a b c d
DSSSD#1 3 4 1 2
DSSSD#2 7 8 5 6

p+n junction FE64s odd numbered


Data analysis assumes

- all LEC ADC data channels with valid ADC offset included (474 of 512 channels)
- no clustering
- no p+n junction side - n+n Ohmic side correlation time gates
- valid LEC events
   0 < p+n junction side multiplicity < 8 
   and
   0 < n+n Ohmic side multiplicity < 8 


Attachments 1-2 - per DSSSD p+n versus n+n multiplicity

Attachment 3 - per DSSSD x versus y

Attachments 4-5 - per DSSSD p+n versus n+n energy (20keV/channel nominal)
                  all combinations of per DSSSD p+n junction and n+n Ohmic energies 
                  with projection of data within window onto x and y axes

                  too many events for natural (U decay series) background
                  off leading diagonal correlations anomalous
                  transverse width of leading diagonal correlation wider than expected - ADC offsets OK?
                 

Attachment 6 - per FEE64 WR timestamp (32.768us/channel)
               FEE64 sync test using pulser data - looks OK

Attachments 7-14 - per FEE64 ADC spectra (5.6keV/channel nominal)
                   note common x/y scale - pulser peak height proxy for peak width

per FEE64 1.8.L pulser peak widths (ch FWHM)

1 11.43
2 16.58
3 12.52
4 17.54
5 9.10
6 12.22
7 17.03
8 16.09

3 of 4 p+n junction FEE64 good (<70keV), 1 of 4 n+n Ohmic FEE64 good - all others < 100keV
Pulser peak indicates noise/gain/offset stable throughout background runs


Attachments 15-22 - per FEE64 ADC spectra (5.6kleV/channel nominal)
                    observe broad peak-like structures (c. 2.8MeV) in first channel of most ASICs? 

Attachment 23 per DSSSD p+n versus n+n energy (20keV/channel nominal)
              valid LEC events
              p+n junction side multiplicity = 1 n+n Ohmic side multiplicity = 1

Attachment 24 p+n FEE64 #0 energy versus each of n+n FEE64s (#1, #3, #5 and #7) energies (20keV/channel nominal)
              valid LEC events
              p+n junction side multiplicity = 1 n+n Ohmic side multiplicity = 1

Attachments 23 & 24 do not unambiguously identify which FEE64s are attached to which DSSSD
                  
Attachment 1: Screenshot_from_2024-11-06_10-16-55.png
Screenshot_from_2024-11-06_10-16-55.png
Attachment 2: Screenshot_from_2024-11-06_10-18-43.png
Screenshot_from_2024-11-06_10-18-43.png
Attachment 3: Screenshot_from_2024-11-06_10-19-38.png
Screenshot_from_2024-11-06_10-19-38.png
Attachment 4: Screenshot_from_2024-11-06_10-21-13.png
Screenshot_from_2024-11-06_10-21-13.png
Attachment 5: Screenshot_from_2024-11-06_10-23-42.png
Screenshot_from_2024-11-06_10-23-42.png
Attachment 6: Screenshot_from_2024-11-06_10-47-04.png
Screenshot_from_2024-11-06_10-47-04.png
Attachment 7: Screenshot_from_2024-11-06_10-49-16.png
Screenshot_from_2024-11-06_10-49-16.png
Attachment 8: Screenshot_from_2024-11-06_10-50-17.png
Screenshot_from_2024-11-06_10-50-17.png
Attachment 9: Screenshot_from_2024-11-06_10-50-59.png
Screenshot_from_2024-11-06_10-50-59.png
Attachment 10: Screenshot_from_2024-11-06_10-51-48.png
Screenshot_from_2024-11-06_10-51-48.png
Attachment 11: Screenshot_from_2024-11-06_10-52-49.png
Screenshot_from_2024-11-06_10-52-49.png
Attachment 12: Screenshot_from_2024-11-06_10-54-12.png
Screenshot_from_2024-11-06_10-54-12.png
Attachment 13: Screenshot_from_2024-11-06_10-55-07.png
Screenshot_from_2024-11-06_10-55-07.png
Attachment 14: Screenshot_from_2024-11-06_10-55-56.png
Screenshot_from_2024-11-06_10-55-56.png
Attachment 15: Screenshot_from_2024-11-06_11-01-47.png
Screenshot_from_2024-11-06_11-01-47.png
Attachment 16: Screenshot_from_2024-11-06_11-02-48.png
Screenshot_from_2024-11-06_11-02-48.png
Attachment 17: Screenshot_from_2024-11-06_11-04-21.png
Screenshot_from_2024-11-06_11-04-21.png
Attachment 18: Screenshot_from_2024-11-06_11-05-09.png
Screenshot_from_2024-11-06_11-05-09.png
Attachment 19: Screenshot_from_2024-11-06_11-06-16.png
Screenshot_from_2024-11-06_11-06-16.png
Attachment 20: Screenshot_from_2024-11-06_11-07-16.png
Screenshot_from_2024-11-06_11-07-16.png
Attachment 21: Screenshot_from_2024-11-06_11-08-07.png
Screenshot_from_2024-11-06_11-08-07.png
Attachment 22: Screenshot_from_2024-11-06_11-09-09.png
Screenshot_from_2024-11-06_11-09-09.png
Attachment 23: Screenshot_from_2024-11-08_03-36-38.png
Screenshot_from_2024-11-08_03-36-38.png
Attachment 24: Screenshot_from_2024-11-08_03-39-13.png
Screenshot_from_2024-11-08_03-39-13.png
  669   Wed Aug 14 12:40:11 2024 JBRepaired DSSSD delivery 14.09.2024

Three BB18-1000 triples AIDAs collected on 14.09.2024

Find attached visual of the wafer and bond wire + factory bias tests accompanying the DSSSDs. elog:669/1 elog:669/2 elog:669/3

Visual inspection carried out showed that bond wires have been fixed + fingerprint on one DSSSD removed and wires repaired. elog:669/4 elog:669/5 elog:669/6 elog:669/7 elog:669/8 elog:669/9 elog:669/10 elog:669/11

DSSSD 1 (defect bias issue 80V): 3208-10 / 3208-18 / 3208-20

DSSSD 2  (3208-6 dysfunctional): 3208-6 / 3208-9 / 3208-16

DSSSD 3 (defect fingerprint): 3131-5 / 3131-10 / 3131-12

 

Attachment 1: doc00418120240814132739.pdf
doc00418120240814132739.pdf
Attachment 2: doc00418020240814132419.pdf
doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf doc00418020240814132419.pdf
Attachment 3: doc00417920240814132325.pdf
doc00417920240814132325.pdf doc00417920240814132325.pdf doc00417920240814132325.pdf
Attachment 4: 20240814_132617.jpg
20240814_132617.jpg
Attachment 5: 20240814_133316.jpg
20240814_133316.jpg
Attachment 6: 20240814_133306.jpg
20240814_133306.jpg
Attachment 7: 20240814_133225.jpg
20240814_133225.jpg
Attachment 8: 20240814_133321.jpg
20240814_133321.jpg
Attachment 9: 20240814_133328.jpg
20240814_133328.jpg
Attachment 10: 20240814_133300.jpg
20240814_133300.jpg
Attachment 11: 20240814_133313.jpg
20240814_133313.jpg
  668   Sun Jul 14 16:52:28 2024 TDS181 R7_38-44
Attachment 1: R7_44.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257387880 ( 1999547.7 Hz)
        Other data format:    4532120 (   35208.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1033 (       8.0 Hz)
                               RESUME:       1034 (       8.0 Hz)
                              SYNC100:      32479 (     252.3 Hz)
                              WR48-63:      32479 (     252.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4465095 (   34687.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     632762 (    4915.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      128.723 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.236                0.000
  1                1.555                0.000
  2                0.113                0.000
  3                0.947                0.000
  4                0.002                0.000
  5                1.755                0.000
  6                0.004                0.000
  7               10.333                0.000
  8                0.007                0.000
  9                2.936                0.000
 10                2.071                0.000
 11                0.002                0.000
 12                0.000                0.000
 13                0.489                0.000
 14                0.000                0.000
 15                1.734                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14350588       7653          0          0         30         30       1869       1869          0       3855          0      17361
  1   13486473       3480          0          0         46         46       1694       1694          0          0          0      37855
  2   14703665    1320670          0          0         15         15       2021       2021          0    1316598          0      23155
  3   19664164       4986          0          0         55         55       2438       2438          0          0          0      30308
  4    5751938     622359          0          0          1          1        800        800          0     620757          0      10363
  5   18754826       4878          0          0        135        135       2304       2304          0          0          0      35466
  6    5749940     466689          0          0          1          1        788        788          0     465111          0      22939
  7   43562502      11323          0          0        235        236       5426       5426          0          0          0     306189
  8    6498781       1588          0          0          3          3        791        791          0          0          0      27083
  9   35613051    1294837          0          0        123        123       4631       4631          0    1285329          0      25629
 10   18394694       5004          0          0        235        235       2267       2267          0          0          0      15231
 11    4718587     774761          0          0          1          1        657        657          0     773445          0      24458
 12    4562825       1124          0          0          0          0        562        562          0          0          0       6576
 13   14049104       3450          0          0         62         62       1663       1663          0          0          0      22623
 14    5429441       1318          0          0          0          0        659        659          0          0          0      16486
 15   32097301       8000          0          0         91         91       3909       3909          0          0          0      11040
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.094s ( 1099.893 blocks/s,  68.743 Mb/s)
Attachment 2: R7_43.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257380554 ( 1952632.9 Hz)
        Other data format:    4539446 (   34438.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1049 (       8.0 Hz)
                               RESUME:       1048 (       8.0 Hz)
                              SYNC100:      32491 (     246.5 Hz)
                              WR48-63:      32491 (     246.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4472367 (   33929.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     637947 (    4839.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      131.812 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.211                0.000
  1                0.644                0.000
  2                0.104                0.000
  3                0.610                0.000
  4                0.001                0.000
  5                1.489                0.000
  6                0.000                0.000
  7               10.407                0.000
  8                0.007                0.000
  9                3.903                0.000
 10                2.105                0.000
 11                0.015                0.000
 12                0.000                0.000
 13                0.359                0.000
 14                0.001                0.000
 15                2.207                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14064394       7614          0          0         30         30       1803       1803          0       3948          0      17314
  1   13179423       3388          0          0         29         29       1665       1665          0          0          0      38151
  2   14557381    1335568          0          0         13         13       1972       1972          0    1331598          0      23475
  3   19617646       4892          0          0         43         43       2403       2403          0          0          0      30324
  4    5648958     616805          0          0          1          1        777        777          0     615249          0      10380
  5   18696180       4932          0          0        131        131       2335       2335          0          0          0      35339
  6    5815232     464164          0          0          0          0        770        770          0     462624          0      23216
  7   44269814      11633          0          0        225        224       5592       5592          0          0          0     313471
  8    6429091       1516          0          0          1          1        757        757          0          0          0      26517
  9   35880468    1300754          0          0        152        152       4574       4574          0    1291302          0      24969
 10   18353794       4866          0          0        246        246       2187       2187          0          0          0      14856
 11    4621196     769080          0          0          4          4        713        713          0     767646          0      24241
 12    4572345       1024          0          0          0          0        512        512          0          0          0       6439
 13   13956451       3570          0          0         56         56       1729       1729          0          0          0      22520
 14    5345915       1318          0          0          1          1        658        658          0          0          0      15800
 15   32372266       8322          0          0        117        117       4044       4044          0          0          0      10935
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.129s ( 1098.565 blocks/s,  68.660 Mb/s)
Attachment 3: R7_42.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257357615 ( 1959593.4 Hz)
        Other data format:    4562385 (   34739.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1086 (       8.3 Hz)
                               RESUME:       1086 (       8.3 Hz)
                              SYNC100:      32490 (     247.4 Hz)
                              WR48-63:      32490 (     247.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4495233 (   34228.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     640512 (    4877.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      131.332 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.320                0.000
  1                0.555                0.000
  2                0.107                0.000
  3                1.008                0.000
  4                0.000                0.000
  5                1.900                0.000
  6                0.000                0.000
  7               10.018                0.000
  8                0.003                0.000
  9                3.911                0.000
 10                2.003                0.000
 11                0.016                0.000
 12                0.000                0.000
 13                0.450                0.000
 14                0.001                0.000
 15                1.832                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13873525       7404          0          0         41         41       1699       1699          0       3924          0      17674
  1   13645773       3570          0          0         26         26       1759       1759          0          0          0      38841
  2   14382089    1335288          0          0         15         15       1902       1902          0    1331454          0      23704
  3   19838847       5258          0          0         58         58       2571       2571          0          0          0      30478
  4    5609330     626216          0          0          1          1        780        780          0     624654          0      10429
  5   18732852       4878          0          0        132        132       2307       2307          0          0          0      35532
  6    5773741     470345          0          0          0          0        796        796          0     468753          0      23367
  7   44302756      11512          0          0        235        235       5521       5521          0          0          0     312043
  8    6624893       1678          0          0          1          1        838        838          0          0          0      26919
  9   35723194    1298446          0          0        149        149       4530       4530          0    1289088          0      25317
 10   18313583       5042          0          0        245        245       2276       2276          0          0          0      15157
 11    4596970     778626          0          0          5          5        628        628          0     777360          0      24418
 12    4585974       1066          0          0          0          0        533        533          0          0          0       6525
 13   13753094       3518          0          0         70         70       1689       1689          0          0          0      22808
 14    5336029       1368          0          0          1          1        683        683          0          0          0      16140
 15   32264965       8170          0          0        107        107       3978       3978          0          0          0      11160
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.191s ( 1096.213 blocks/s,  68.513 Mb/s)
Attachment 4: R7_41.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257421108 ( 1939002.8 Hz)
        Other data format:    4498892 (   33887.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1059 (       8.0 Hz)
                               RESUME:       1059 (       8.0 Hz)
                              SYNC100:      32494 (     244.8 Hz)
                              WR48-63:      32494 (     244.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4431786 (   33382.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     626091 (    4716.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      132.760 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.208                0.000
  1                0.426                0.000
  2                0.135                0.000
  3                0.540                0.000
  4                0.004                0.000
  5                1.529                0.000
  6                0.001                0.000
  7               11.561                0.000
  8                0.001                0.000
  9                3.166                0.000
 10                1.981                0.000
 11                0.010                0.000
 12                0.003                0.000
 13                0.435                0.000
 14                0.001                0.000
 15                2.192                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14125854       7603          0          0         34         34       1780       1780          0       3975          0      16932
  1   12999507       3346          0          0         27         27       1646       1646          0          0          0      37605
  2   14549105    1330849          0          0         16         16       1993       1993          0    1326831          0      22570
  3   19588079       5104          0          0         44         44       2508       2508          0          0          0      29463
  4    5623374     602489          0          0          1          1        771        771          0     600945          0      10031
  5   18635917       5070          0          0        145        145       2390       2390          0          0          0      34577
  6    5774011     454485          0          0          1          1        755        755          0     452973          0      22358
  7   43927148      11232          0          0        252        252       5364       5364          0          0          0     309324
  8    6481984       1536          0          0          1          1        767        767          0          0          0      25848
  9   36293972    1305814          0          0        134        134       4617       4617          0    1296312          0      24663
 10   18484729       5190          0          0        226        226       2369       2369          0          0          0      14628
 11    4592277     752114          0          0          2          2        680        680          0     750750          0      23479
 12    4565033       1142          0          0          2          2        569        569          0          0          0       6273
 13   13975352       3570          0          0         64         64       1721       1721          0          0          0      21912
 14    5319152       1192          0          0          1          1        595        595          0          0          0      15594
 15   32485614       8156          0          0        109        109       3969       3969          0          0          0      10834
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.234s ( 1094.602 blocks/s,  68.413 Mb/s)
Attachment 5: R7_40.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257446831 ( 1940911.1 Hz)
        Other data format:    4473169 (   33723.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1093 (       8.2 Hz)
                               RESUME:       1093 (       8.2 Hz)
                              SYNC100:      32494 (     245.0 Hz)
                              WR48-63:      32494 (     245.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4405995 (   33217.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     622088 (    4690.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      132.642 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.216                0.000
  1                0.343                0.000
  2                0.126                0.000
  3                0.759                0.000
  4                0.006                0.000
  5                1.457                0.000
  6                0.000                0.000
  7               11.340                0.000
  8                0.008                0.000
  9                3.704                0.000
 10                2.066                0.000
 11                0.009                0.000
 12                0.000                0.000
 13                0.422                0.000
 14                0.007                0.000
 15                1.901                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14192134       7643          0          0         32         32       1802       1802          0       3975          0      16831
  1   12692374       3340          0          0         21         21       1649       1649          0          0          0      37336
  2   14587574    1324651          0          0         22         22       2017       2017          0    1320573          0      22631
  3   19453349       5132          0          0         57         57       2509       2509          0          0          0      29511
  4    5663707     601044          0          0          2          2        715        715          0     599610          0       9909
  5   18656910       4898          0          0        117        117       2332       2332          0          0          0      34162
  6    5798018     454961          0          0          0          0        757        757          0     453447          0      22350
  7   44082007      11418          0          0        263        263       5446       5446          0          0          0     306534
  8    6461137       1572          0          0          4          4        782        782          0          0          0      25967
  9   36131717    1290933          0          0        150        150       4584       4584          0    1281465          0      24188
 10   18542819       5234          0          0        249        249       2368       2368          0          0          0      14717
 11    4581074     748195          0          0          3          3        632        632          0     746925          0      23542
 12    4576883       1108          0          0          0          0        554        554          0          0          0       6304
 13   14043798       3508          0          0         66         66       1688       1688          0          0          0      21907
 14    5310480       1328          0          0          2          2        662        662          0          0          0      15552
 15   32672850       8204          0          0        105        105       3997       3997          0          0          0      10647
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.266s ( 1093.433 blocks/s,  68.340 Mb/s)
Attachment 6: R7_39.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257666264 ( 1899576.9 Hz)
        Other data format:    4253736 (   31359.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1109 (       8.2 Hz)
                               RESUME:       1109 (       8.2 Hz)
                              SYNC100:      32506 (     239.6 Hz)
                              WR48-63:      32506 (     239.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4186506 (   30863.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     603733 (    4450.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      135.644 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.315                0.000
  1                0.465                0.000
  2                0.152                0.000
  3                0.477                0.000
  4                0.000                0.000
  5                1.410                0.000
  6                0.000                0.000
  7               10.830                0.000
  8                0.006                0.000
  9                3.516                0.000
 10                2.096                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.504                0.000
 14                0.001                0.000
 15                2.318                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14135604       7717          0          0         45         45       1787       1787          0       4053          0      15313
  1   11750051       2976          0          0         28         28       1460       1460          0          0          0      33988
  2   14645076    1299621          0          0         18         18       2082       2082          0    1295421          0      20497
  3   19028955       4938          0          0         42         42       2427       2427          0          0          0      26749
  4    5601003     542795          0          0          0          0        778        778          0     541239          0       8896
  5   18444566       4954          0          0        128        128       2349       2349          0          0          0      31140
  6    5705092     411813          0          0          1          1        713        713          0     410385          0      20283
  7   44751821      11668          0          0        247        247       5587       5587          0          0          0     317392
  8    6384929       1562          0          0          3          3        778        778          0          0          0      23462
  9   36840506    1271195          0          0        144        144       4618       4618          0    1261671          0      22142
 10   18869011       5166          0          0        245        245       2338       2338          0          0          0      13407
 11    4447340     674973          0          0          1          1        617        617          0     673737          0      21305
 12    4493746       1090          0          0          0          0        545        545          0          0          0       5562
 13   14105097       3536          0          0         80         80       1688       1688          0          0          0      19880
 14    5253775       1242          0          0          1          1        620        620          0          0          0      14101
 15   33209692       8490          0          0        126        126       4119       4119          0          0          0       9616
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.145s ( 1097.976 blocks/s,  68.624 Mb/s)
Attachment 7: R7_38.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258699123 ( 1711360.3 Hz)
        Other data format:    3220878 (   21306.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        926 (       6.1 Hz)
                               RESUME:        926 (       6.1 Hz)
                              SYNC100:      32563 (     215.4 Hz)
                              WR48-63:      32563 (     215.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3153900 (   20863.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     485537 (    3212.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      151.166 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.249                0.000
  1                0.120                0.000
  2                0.066                0.000
  3                0.354                0.000
  4                0.003                0.000
  5                0.536                0.000
  6                0.000                0.000
  7                6.425                0.000
  8                0.006                0.000
  9                3.101                0.000
 10                1.800                0.000
 11                0.004                0.000
 12                0.000                0.000
 13                0.331                0.000
 14                0.001                0.000
 15                1.796                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14033681       8147          0          0         36         36       1771       1771          0       4533          0       6714
  1    7323901       1916          0          0          9          9        949        949          0          0          0      15627
  2   14901799    1219051          0          0         10         10       2080       2080          0    1214871          0       9105
  3   17459053       4614          0          0         34         34       2273       2273          0          0          0      11881
  4    5291930     245124          0          0          1          1        734        734          0     243654          0       4222
  5   16613617       4290          0          0         76         76       2069       2069          0          0          0      14390
  6    5335939     196741          0          0          0          0        710        710          0     195321          0       9119
  7   48661913      12390          0          0        201        201       5994       5994          0          0          0     356859
  8    5678372       1424          0          0          2          2        710        710          0          0          0      10389
  9   40228814    1200218          0          0        144        144       5116       5116          0    1189698          0       9803
 10   20091293       5418          0          0        251        251       2458       2458          0          0          0       5747
 11    3741989     306779          0          0          1          1        477        477          0     305823          0       9660
 12    3923680        940          0          0          0          0        470        470          0          0          0       2649
 13   14468402       3682          0          0         54         54       1787       1787          0          0          0       8802
 14    4770705       1114          0          0          1          1        556        556          0          0          0       6164
 15   36174035       9030          0          0        106        106       4409       4409          0          0          0       4406
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.168s ( 1097.094 blocks/s,  68.568 Mb/s)
  667   Sun Jul 14 14:37:53 2024 TDS181 R6_375-380
Attachment 1: R6_380.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258540051 ( 2422222.2 Hz)
        Other data format:    3379949 (   31666.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1735 (      16.3 Hz)
                               RESUME:       1737 (      16.3 Hz)
                              SYNC100:      32396 (     303.5 Hz)
                              WR48-63:      32396 (     303.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3311685 (   31026.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     517373 (    4847.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      106.737 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.790                0.000
  1                8.947                0.000
  2                0.508                0.000
  3               19.756                0.000
  4                4.785                0.000
  5               20.920                0.000
  6                0.022                0.000
  7               25.352                0.000
  8                0.008                0.000
  9                3.185                0.000
 10                2.367                0.000
 11                0.358                0.000
 12                2.272                0.000
 13                0.389                0.000
 14                0.017                0.000
 15                2.304                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14698695       7082          0          0         57         57       1903       1903          0       3162          0      34241
  1   16000888       4254          0          0        104        104       2023       2023          0          0          0      42387
  2   14179598    1061970          0          0         24         24       1878       1878          0    1058166          0      16158
  3   21475534       6065          0          0        356        357       2676       2676          0          0          0      27190
  4   13248260     388594          0          0         86         86       1647       1647          0     385128          0      11106
  5   22463599       5966          0          0        228        228       2755       2755          0          0          0      27752
  6    5791719     322542          0          0          5          5        778        778          0     320976          0      14590
  7   35198362       9279          0          0        282        283       4357       4357          0          0          0     182456
  8    5823243       1532          0          0          2          2        764        764          0          0          0      15809
  9   28506851    1036505          0          0        134        134       3608       3608          0    1029021          0      13553
 10   15170339       4094          0          0        198        198       1849       1849          0          0          0       7980
 11    9668170     517738          0          0         28         28       1225       1225          0     515232          0      45139
 12   10627809       2786          0          0         97         97       1296       1296          0          0          0      10320
 13   13616594       3424          0          0         42         42       1670       1670          0          0          0      24397
 14    5270456       1294          0          0          5          5        642        642          0          0          0       8930
 15   26799934       6824          0          0         87         87       3325       3325          0          0          0      35365
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.965s ( 1067.918 blocks/s,  66.745 Mb/s)
Attachment 2: R6_379.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258492867 ( 2488518.1 Hz)
        Other data format:    3427133 (   32993.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1669 (      16.1 Hz)
                               RESUME:       1667 (      16.0 Hz)
                              SYNC100:      32386 (     311.8 Hz)
                              WR48-63:      32386 (     311.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3359025 (   32337.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     534041 (    5141.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      103.874 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.531                0.000
  1                9.105                0.000
  2                0.409                0.000
  3               20.901                0.000
  4                6.817                0.000
  5               21.213                0.000
  6                0.044                0.000
  7               25.524                0.000
  8                0.012                0.000
  9                2.155                0.000
 10                2.654                0.000
 11                0.254                0.000
 12                3.206                0.000
 13                0.405                0.000
 14                0.021                0.000
 15                1.675                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14465712       6849          0          0         46         46       1829       1829          0       3099          0      35543
  1   16450883       4372          0          0         97         97       2089       2089          0          0          0      45383
  2   14055542    1052711          0          0         21         21       1792       1792          0    1049085          0      16931
  3   21598001       6069          0          0        353        352       2682       2682          0          0          0      28874
  4   13375374     396382          0          0         93         93       1673       1673          0     392850          0      11316
  5   23169470       6376          0          0        222        222       2966       2966          0          0          0      29587
  6    5868310     338835          0          0          6          6        756        756          0     337311          0      15349
  7   34787878       9109          0          0        284        283       4271       4271          0          0          0     179444
  8    5913627       1550          0          0          4          4        771        771          0          0          0      16483
  9   28000125    1037778          0          0         94         94       3590       3590          0    1030410          0      14411
 10   14658992       4034          0          0        206        206       1811       1811          0          0          0       8453
 11    9978683     548866          0          0         21         21       1277       1277          0     546270          0      48576
 12   11124676       2942          0          0        108        108       1363       1363          0          0          0      10622
 13   13313729       3352          0          0         38         38       1638       1638          0          0          0      25010
 14    5305479       1290          0          0          7          7        638        638          0          0          0       9613
 15   26426386       6618          0          0         69         69       3240       3240          0          0          0      38446
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.004s ( 1066.528 blocks/s,  66.658 Mb/s)
Attachment 3: R6_378.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258502011 ( 2458827.2 Hz)
        Other data format:    3417989 (   32511.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1749 (      16.6 Hz)
                               RESUME:       1749 (      16.6 Hz)
                              SYNC100:      32392 (     308.1 Hz)
                              WR48-63:      32392 (     308.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3349707 (   31861.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     532400 (    5064.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      105.132 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.679                0.000
  1                9.249                0.000
  2                0.786                0.000
  3               21.653                0.000
  4                5.446                0.000
  5               20.569                0.000
  6                0.018                0.000
  7               25.965                0.000
  8                0.023                0.000
  9                3.527                0.000
 10                3.084                0.000
 11                0.578                0.000
 12                2.211                0.000
 13                0.401                0.000
 14                0.029                0.000
 15                2.039                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14837825       7065          0          0         54         54       1908       1908          0       3141          0      35149
  1   15971414       4190          0          0         90         90       2005       2005          0          0          0      44340
  2   13973975    1055388          0          0         28         28       1877       1877          0    1051578          0      16603
  3   21360293       6062          0          0        359        359       2672       2672          0          0          0      27349
  4   13445710     405480          0          0         78         78       1722       1722          0     401880          0      11761
  5   22822103       6008          0          0        219        219       2785       2785          0          0          0      29227
  6    5869439     336469          0          0          3          3        764        764          0     334935          0      15220
  7   34904209       9014          0          0        288        288       4219       4219          0          0          0     183752
  8    5764878       1344          0          0          3          3        669        669          0          0          0      16353
  9   28081264    1028257          0          0        128        128       3666       3666          0    1020669          0      13917
 10   14858837       4056          0          0        228        228       1800       1800          0          0          0       8182
 11   10178261     540198          0          0         44         44       1303       1303          0     537504          0      47495
 12   10803258       2976          0          0        104        104       1384       1384          0          0          0      11101
 13   13631305       3368          0          0         38         38       1646       1646          0          0          0      24976
 14    5355037       1370          0          0         10         10        675        675          0          0          0       9380
 15   26644203       6744          0          0         75         75       3297       3297          0          0          0      37595
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.863s ( 1036.831 blocks/s,  64.802 Mb/s)
Attachment 4: R6_377.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258552134 ( 2473392.2 Hz)
        Other data format:    3367866 (   32218.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1756 (      16.8 Hz)
                               RESUME:       1758 (      16.8 Hz)
                              SYNC100:      32389 (     309.8 Hz)
                              WR48-63:      32389 (     309.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3299574 (   31564.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     527301 (    5044.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      104.533 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.597                0.000
  1                8.636                0.000
  2                0.904                0.000
  3               20.863                0.000
  4                6.343                0.000
  5               20.105                0.000
  6                0.078                0.000
  7               25.706                0.000
  8                0.032                0.000
  9                3.757                0.000
 10                2.723                0.000
 11                0.441                0.000
 12                2.624                0.000
 13                0.350                0.000
 14                0.041                0.000
 15                2.134                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14829940       6930          0          0         43         43       1859       1859          0       3126          0      34968
  1   16084855       4240          0          0         86         86       2034       2034          0          0          0      44487
  2   14095291    1043346          0          0         25         25       1901       1901          0    1039494          0      16177
  3   21497704       5946          0          0        351        351       2622       2622          0          0          0      27780
  4   13645755     392320          0          0         90         90       1745       1745          0     388650          0      11516
  5   22699356       6082          0          0        212        212       2829       2829          0          0          0      28940
  6    5799289     329603          0          0          9          9        757        757          0     328071          0      14660
  7   34611640       9059          0          0        283        284       4246       4246          0          0          0     182416
  8    5904403       1412          0          0          5          5        701        701          0          0          0      16059
  9   27962268    1015369          0          0        138        139       3606       3606          0    1007880          0      13650
 10   14955138       4224          0          0        234        234       1878       1878          0          0          0       8242
 11    9968144     534917          0          0         35         35       1247       1247          0     532353          0      46585
 12   11007856       2816          0          0        109        109       1299       1299          0          0          0      10892
 13   13574135       3480          0          0         32         32       1708       1708          0          0          0      24567
 14    5407517       1306          0          0          8          8        645        645          0          0          0       9376
 15   26508843       6816          0          0         96         96       3312       3312          0          0          0      36986
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.332s ( 1054.990 blocks/s,  65.937 Mb/s)
Attachment 5: R6_376.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258492402 ( 2497580.9 Hz)
        Other data format:    3427598 (   33117.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1769 (      17.1 Hz)
                               RESUME:       1767 (      17.1 Hz)
                              SYNC100:      32385 (     312.9 Hz)
                              WR48-63:      32385 (     312.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3359292 (   32457.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     535580 (    5174.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      103.497 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.776                0.000
  1                9.861                0.000
  2                0.678                0.000
  3               21.902                0.000
  4                6.683                0.000
  5               20.300                0.000
  6                0.016                0.000
  7               26.525                0.000
  8                0.027                0.000
  9                3.024                0.000
 10                2.649                0.000
 11                0.458                0.000
 12                2.900                0.000
 13                0.387                0.000
 14                0.048                0.000
 15                2.584                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14764877       6954          0          0         56         56       1876       1876          0       3090          0      36135
  1   16480843       4336          0          0        118        118       2050       2050          0          0          0      45658
  2   14252286    1048007          0          0         23         23       1904       1904          0    1044153          0      16692
  3   21564846       6026          0          0        350        350       2663       2663          0          0          0      28647
  4   13304736     404308          0          0         88         88       1702       1702          0     400728          0      11809
  5   23013714       6376          0          0        231        231       2957       2957          0          0          0      30027
  6    5873061     342063          0          0          3          3        780        780          0     340497          0      15123
  7   34298981       8993          0          0        299        298       4198       4198          0          0          0     178104
  8    5866367       1500          0          0          3          3        747        747          0          0          0      16863
  9   27866733    1026338          0          0        122        121       3511       3511          0    1019073          0      14551
 10   14765796       4022          0          0        190        190       1821       1821          0          0          0       8481
 11   10252313     554457          0          0         32         32       1321       1321          0     551751          0      48407
 12   11025718       2958          0          0        115        115       1364       1364          0          0          0      11098
 13   13565855       3326          0          0         35         35       1628       1628          0          0          0      26079
 14    5404814       1304          0          0         10         10        642        642          0          0          0       9715
 15   26191462       6630          0          0         94         94       3221       3221          0          0          0      38191
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.238s ( 1058.261 blocks/s,  66.141 Mb/s)
Attachment 6: R6_375.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258498122 ( 2532033.6 Hz)
        Other data format:    3421878 (   33517.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1769 (      17.3 Hz)
                               RESUME:       1769 (      17.3 Hz)
                              SYNC100:      32380 (     317.2 Hz)
                              WR48-63:      32380 (     317.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3353580 (   32848.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     533348 (    5224.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      102.091 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.722                0.000
  1                9.219                0.000
  2                0.540                0.000
  3               21.422                0.000
  4                4.718                0.000
  5               21.434                0.000
  6                0.023                0.000
  7               26.932                0.000
  8                0.014                0.000
  9                3.140                0.000
 10                2.560                0.000
 11                0.453                0.000
 12                2.280                0.000
 13                0.408                0.000
 14                0.012                0.000
 15                2.073                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14893366       6777          0          0         60         60       1809       1809          0       3039          0      36151
  1   16620608       4354          0          0        112        112       2065       2065          0          0          0      45709
  2   13854193    1042012          0          0         29         29       1893       1893          0    1038168          0      16872
  3   21513605       6166          0          0        352        352       2731       2731          0          0          0      28236
  4   13835719     417225          0          0         86         86       1762       1762          0     413529          0      11794
  5   22712864       6078          0          0        213        213       2826       2826          0          0          0      29593
  6    5842394     341982          0          0          7          7        740        740          0     340488          0      15332
  7   33822775       8894          0          0        271        271       4176       4176          0          0          0     176325
  8    5896518       1406          0          0          3          3        700        700          0          0          0      16319
  9   27703752    1015898          0          0        111        111       3577       3577          0    1008522          0      13968
 10   14877004       4052          0          0        234        234       1792       1792          0          0          0       8511
 11   10500301     552528          0          0         30         30       1317       1317          0     549834          0      49042
 12   11072905       2914          0          0        119        119       1338       1338          0          0          0      10998
 13   13651284       3526          0          0         44         44       1719       1719          0          0          0      25633
 14    5433744       1396          0          0          4          4        694        694          0          0          0       9657
 15   26267090       6670          0          0         94         94       3241       3241          0          0          0      39208
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.184s ( 1060.179 blocks/s,  66.261 Mb/s)
  666   Mon Jun 17 02:13:21 2024 TDMonday 17 June
03.11 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      per FEE64 Rate spectra - attachment 4

05.33 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      per FEE64 Rate spectra - attachment 8

      per FEE64 1.8.L spectra - attachments 9-11
       aida09 pulser walkthrough 60 ch FWHM

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-14

      Merger, TapeServer etc - attachments 15-16

05.40 analysis data file R9_1042 - attachment 17

      Check ASIC load x2

      aida02 zero rate
      restart Merger

05.55 Data file S181/R10
      Pulser walkthrough

      All histograms zero'd

      BNC PB-5
      Amplitudes 10.0-1.0 @ 1.0 step
      Attenuation x10
      Polarity +
      tau_d 1ms
      Frequency 25Hz

      per p+n FEE64 1.8.L spectra - attachment 18

      per FEE64 Rate spectra - attachment 19

06.43 TapeServer no storage mode

09.14 Data file S181/R11
      Pulser walkthrough

      All histograms zero'd

      BNC PB-5
      Amplitudes 10.0-1.0 @ 1.0 step
      Attenuation x10
      Polarity -
      tau_d 1ms
      Frequency 25Hz

09.52 TapeServer no storage mode

      per n+n FEE64 1.8.L spectra - attachment 20

15.18 DEFGAS/FATIMA arrays moved downstream to permit installation of calibration source at centre of each array - increased distance to AIDA/bPlas => lower rate


15.16 DSSSD bias & leakage current OK - attachments 21-22

      FEE64 temperatures OK - attachment 23

      ADC data item stats OK - attachment 24
       aida02 zero rate

      per FEE64 Rate spectra - attachment 25

      Merger restart
      ADC data item stats OK - attachment 26
       7x < 20k, max 207k aida16

15.54 bPlast debiased.


16.29 Data file S181/R12
      bPlas OFF but remiains cabled/connected/grounded

16.30 DSSSD bias & leakage current OK - attachment 27

      FEE64 temperatures OK - attachment 28

      ADC data item stats OK - attachment 29

      per FEE64 Rate spectra - attachment 30

      per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - see attachments 31-39
       note - not all FEE64s updated spectra for all settings

      ADC data item stats OK - attachment 40
       rates change cf. attachment 29
       sources installed/moved/removed, people moving around platform?

      per FEE64 Rate spectra - attachment 41

16.57 DSSSD bias OFF
      FEE64 power OFF

16.34 analysis data file S181/R12_7
      max deadtime 2.2% (aida08)
      ADC data rate 1.330M, HEC data rate 2.1k

16.52 analysis data file S181/R12_12
      max deadtime 1.7% (aida08)
      ADC data rate 945k, HEC data rate 1.8k
Attachment 1: Screenshot_from_2024-06-17_03-11-38.png
Screenshot_from_2024-06-17_03-11-38.png
Attachment 2: Screenshot_from_2024-06-17_03-12-05.png
Screenshot_from_2024-06-17_03-12-05.png
Attachment 3: Screenshot_from_2024-06-17_03-12-29.png
Screenshot_from_2024-06-17_03-12-29.png
Attachment 4: Screenshot_from_2024-06-17_03-12-53.png
Screenshot_from_2024-06-17_03-12-53.png
Attachment 5: Screenshot_from_2024-06-17_05-33-18.png
Screenshot_from_2024-06-17_05-33-18.png
Attachment 6: Screenshot_from_2024-06-17_05-33-47.png
Screenshot_from_2024-06-17_05-33-47.png
Attachment 7: Screenshot_from_2024-06-17_05-34-22.png
Screenshot_from_2024-06-17_05-34-22.png
Attachment 8: Screenshot_from_2024-06-17_05-34-53.png
Screenshot_from_2024-06-17_05-34-53.png
Attachment 9: Screenshot_from_2024-06-17_05-36-26.png
Screenshot_from_2024-06-17_05-36-26.png
Attachment 10: Screenshot_from_2024-06-17_05-38-40.png
Screenshot_from_2024-06-17_05-38-40.png
Attachment 11: Screenshot_from_2024-06-17_05-39-53.png
Screenshot_from_2024-06-17_05-39-53.png
Attachment 12: Screenshot_from_2024-06-17_05-40-29.png
Screenshot_from_2024-06-17_05-40-29.png
Attachment 13: Screenshot_from_2024-06-17_05-41-12.png
Screenshot_from_2024-06-17_05-41-12.png
Attachment 14: Screenshot_from_2024-06-17_05-42-07.png
Screenshot_from_2024-06-17_05-42-07.png
Attachment 15: Screenshot_from_2024-06-17_05-42-35.png
Screenshot_from_2024-06-17_05-42-35.png
Attachment 16: Screenshot_from_2024-06-17_05-43-06.png
Screenshot_from_2024-06-17_05-43-06.png
Attachment 17: R9_1042
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259119494 ( 1239652.5 Hz)
        Other data format:    2800506 (   13397.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        509 (       2.4 Hz)
                               RESUME:        509 (       2.4 Hz)
                              SYNC100:      32779 (     156.8 Hz)
                              WR48-63:      32779 (     156.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2733930 (   13079.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     425638 (    2036.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      209.026 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.013                0.000
  1                0.000                0.000
  2                0.006                0.000
  3                0.367                0.000
  4                0.000                0.000
  5                0.343                0.000
  6                0.000                0.000
  7                4.268                0.000
  8                0.000                0.000
  9                2.321                0.000
 10                0.126                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.030                0.000
 14                0.000                0.000
 15                1.478                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9607668       8795          0          0         21         21       1240       1240          0       6273          0          0
  1     568341        148          0          0          0          0         74         74          0          0          0          0
  2   11592594    1368472          0          0          2          2       1710       1710          0    1365048          0          0
  3   20057241       5226          0          0         32         32       2581       2581          0          0          0          0
  4    4038921       1042          0          0          0          0        521        521          0          0          0          0
  5   16834727       4288          0          0         27         27       2117       2117          0          0          0          0
  6    2895864      13265          0          0          0          0        361        361          0      12543          0          0
  7   58542191      15052          0          0        171        171       7355       7355          0          0          0     425638
  8    6363514       1604          0          0          0          0        802        802          0          0          0          0
  9   47066112    1362366          0          0        140        140       6010       6010          0    1350066          0          0
 10   19083458       4678          0          0         24         24       2315       2315          0          0          0          0
 11    2437006        602          0          0          0          0        301        301          0          0          0          0
 12    3839787        956          0          0          0          0        478        478          0          0          0          0
 13   12055396       3054          0          0         12         12       1515       1515          0          0          0          0
 14    3699941        872          0          0          0          0        436        436          0          0          0          0
 15   40436733      10086          0          0         80         80       4963       4963          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   69.695s (  459.141 blocks/s,  28.696 Mb/s)
Attachment 18: Screenshot_from_2024-06-17_06-44-16.png
Screenshot_from_2024-06-17_06-44-16.png
Attachment 19: Screenshot_from_2024-06-17_06-45-49.png
Screenshot_from_2024-06-17_06-45-49.png
Attachment 20: Screenshot_from_2024-06-17_09-53-53.png
Screenshot_from_2024-06-17_09-53-53.png
Attachment 21: Screenshot_from_2024-06-17_15-17-20.png
Screenshot_from_2024-06-17_15-17-20.png
Attachment 22: Screenshot_from_2024-06-17_15-16-47.png
Screenshot_from_2024-06-17_15-16-47.png
Attachment 23: Screenshot_from_2024-06-17_15-17-46.png
Screenshot_from_2024-06-17_15-17-46.png
Attachment 24: Screenshot_from_2024-06-17_15-18-09.png
Screenshot_from_2024-06-17_15-18-09.png
Attachment 25: Screenshot_from_2024-06-17_15-18-54.png
Screenshot_from_2024-06-17_15-18-54.png
Attachment 26: Screenshot_from_2024-06-17_15-20-50.png
Screenshot_from_2024-06-17_15-20-50.png
Attachment 27: Screenshot_from_2024-06-17_16-30-26.png
Screenshot_from_2024-06-17_16-30-26.png
Attachment 28: Screenshot_from_2024-06-17_16-30-55.png
Screenshot_from_2024-06-17_16-30-55.png
Attachment 29: Screenshot_from_2024-06-17_16-31-26.png
Screenshot_from_2024-06-17_16-31-26.png
Attachment 30: Screenshot_from_2024-06-17_16-31-48.png
Screenshot_from_2024-06-17_16-31-48.png
Attachment 31: Screenshot_from_2024-06-17_16-33-21.png
Screenshot_from_2024-06-17_16-33-21.png
Attachment 32: Screenshot_from_2024-06-17_16-34-32.png
Screenshot_from_2024-06-17_16-34-32.png
Attachment 33: Screenshot_from_2024-06-17_16-36-35.png
Screenshot_from_2024-06-17_16-36-35.png
Attachment 34: Screenshot_from_2024-06-17_16-37-33.png
Screenshot_from_2024-06-17_16-37-33.png
Attachment 35: Screenshot_from_2024-06-17_16-38-14.png
Screenshot_from_2024-06-17_16-38-14.png
Attachment 36: Screenshot_from_2024-06-17_16-40-21.png
Screenshot_from_2024-06-17_16-40-21.png
Attachment 37: Screenshot_from_2024-06-17_16-41-18.png
Screenshot_from_2024-06-17_16-41-18.png
Attachment 38: Screenshot_from_2024-06-17_16-43-19.png
Screenshot_from_2024-06-17_16-43-19.png
Attachment 39: Screenshot_from_2024-06-17_16-44-24.png
Screenshot_from_2024-06-17_16-44-24.png
Attachment 40: Screenshot_from_2024-06-17_16-45-19.png
Screenshot_from_2024-06-17_16-45-19.png
Attachment 41: Screenshot_from_2024-06-17_16-46-32.png
Screenshot_from_2024-06-17_16-46-32.png
Attachment 42: R12_7
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258100864 ( 1330189.9 Hz)
        Other data format:    3819136 (   19682.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        522 (       2.7 Hz)
                               RESUME:        522 (       2.7 Hz)
                              SYNC100:      32723 (     168.6 Hz)
                              WR48-63:      32723 (     168.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3752646 (   19340.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     409704 (    2111.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      194.033 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.121                0.000
  1                0.000                0.000
  2                0.050                0.000
  3                0.191                0.000
  4                0.010                0.000
  5                0.209                0.000
  6                0.000                0.000
  7                4.225                0.000
  8                0.000                0.000
  9                2.072                0.000
 10                0.065                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.145                0.000
 14                0.002                0.000
 15                1.159                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13428500       9208          0          0         38         38       1659       1659          0       5814          0          0
  1    4595656       1164          0          0          0          0        582        582          0          0          0          0
  2   13843408    1881338          0          0         11         11       1937       1937          0    1877442          0          0
  3   19769270       5066          0          0         33         33       2500       2500          0          0          0          0
  4    5034345       1228          0          0          4          4        610        610          0          0          0          0
  5   19195180       4990          0          0         40         40       2455       2455          0          0          0          0
  6    5179175      12949          0          0          0          0        653        653          0      11643          0          0
  7   53830866      13806          0          0        173        173       6730       6730          0          0          0     409704
  8    6018644       1564          0          0          0          0        782        782          0          0          0          0
  9   41936601    1868943          0          0        126        126       5472       5472          0    1857747          0          0
 10   16378984       4032          0          0         16         16       2000       2000          0          0          0          0
 11    2633344        646          0          0          0          0        323        323          0          0          0          0
 12    1918841        428          0          0          0          0        214        214          0          0          0          0
 13   12464437       2916          0          0         14         14       1444       1444          0          0          0          0
 14    3882711       1026          0          0          1          1        512        512          0          0          0          0
 15   37990902       9832          0          0         66         66       4850       4850          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   60.555s (  528.448 blocks/s,  33.028 Mb/s)
Attachment 43: R12_12
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       7946
          ADC data format:   63712775 (  945168.0 Hz)
        Other data format:    1325235 (   19659.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        128 (       1.9 Hz)
                               RESUME:        128 (       1.9 Hz)
                              SYNC100:       8197 (     121.6 Hz)
                              WR48-63:       8197 (     121.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1308585 (   19412.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     119370 (    1770.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       67.409 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.043                0.000
  4                0.000                0.000
  5                0.026                0.000
  6                0.000                0.000
  7                1.174                0.000
  8                0.000                0.000
  9                0.504                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.224                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     896977       2260          0          0          0          0        119        119          0       2022          0          0
  1    1331618        334          0          0          0          0        167        167          0          0          0          0
  2    2033117     654263          0          0          0          0        346        346          0     653571          0          0
  3    6066546       1620          0          0         14         14        796        796          0          0          0          0
  4     632291        132          0          0          0          0         66         66          0          0          0          0
  5    5664311       1468          0          0         11         11        723        723          0          0          0          0
  6    1249421       4352          0          0          0          0        154        154          0       4044          0          0
  7   16999500       4406          0          0         59         59       2144       2144          0          0          0     119370
  8      43320         14          0          0          0          0          7          7          0          0          0          0
  9   12410393     652234          0          0         26         26       1617       1617          0     648948          0          0
 10    3213140        792          0          0          0          0        396        396          0          0          0          0
 11     278998         64          0          0          0          0         32         32          0          0          0          0
 12     433582        116          0          0          0          0         58         58          0          0          0          0
 13    1869315        430          0          0          0          0        215        215          0          0          0          0
 14     612732        128          0          0          0          0         64         64          0          0          0          0
 15    9977514       2622          0          0         18         18       1293       1293          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   14.254s (  557.461 blocks/s,  34.841 Mb/s)
  665   Sun Jun 16 03:11:27 2024 TDSunday 16 June
04.08 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      per FEE64 Rate spectra - attachment 4


07.52 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      per FEE64 Rate spectra - attachment 8

      per p+n FEE64 1.8.L spectra - attachments 9-10
       aida09 pulser peak width 59 ch FWHM


15.01 DSSSD bias & leakage current OK - attachment 11

      FEE64 temperatures OK - attachment 12

      ADC data item stats OK - attachment 13

      per p+n FEE64 1.8.L spectra - attachments 14-15
       aida09 pulser peak width 59 ch FWHM
       aida05 pulser peak width 68 ch FWHM
       aida12 pulser peak width 71 ch FWHM


      per FEE64 Rate spectra - attachment 16

      Merger, TapeSever etc - attachment 17
       no data transfer to MBS - MBS down?

15.06 analysis data file R9_747 - attachment 18
       max dead time 4.3% aida08, all other FEE64s < 2%
       ADC data rate 1.667M, HEC data rate 2.3k 


19.12 DSSSD bias & leakage current OK - attachment 19

      FEE64 temperatures OK - attachment 20

      ADC data item stats OK - attachment 21

      per FEE64 Rate spectra - attachment 22

23.34 DSSSD bias & leakage current OK - attachment 23

      FEE64 temperatures OK - attachment 24

      ADC data item stats OK - attachment 25

      per FEE64 Rate spectra - attachment 26
Attachment 1: Screenshot_from_2024-06-16_04-10-35.png
Screenshot_from_2024-06-16_04-10-35.png
Attachment 2: Screenshot_from_2024-06-16_04-09-07.png
Screenshot_from_2024-06-16_04-09-07.png
Attachment 3: Screenshot_from_2024-06-16_04-09-41.png
Screenshot_from_2024-06-16_04-09-41.png
Attachment 4: Screenshot_from_2024-06-16_04-10-21.png
Screenshot_from_2024-06-16_04-10-21.png
Attachment 5: Screenshot_from_2024-06-16_07-52-49.png
Screenshot_from_2024-06-16_07-52-49.png
Attachment 6: Screenshot_from_2024-06-16_07-53-20.png
Screenshot_from_2024-06-16_07-53-20.png
Attachment 7: Screenshot_from_2024-06-16_07-53-47.png
Screenshot_from_2024-06-16_07-53-47.png
Attachment 8: Screenshot_from_2024-06-16_07-54-42.png
Screenshot_from_2024-06-16_07-54-42.png
Attachment 9: Screenshot_from_2024-06-16_07-59-28.png
Screenshot_from_2024-06-16_07-59-28.png
Attachment 10: Screenshot_from_2024-06-16_08-01-35.png
Screenshot_from_2024-06-16_08-01-35.png
Attachment 11: Screenshot_from_2024-06-16_15-01-23.png
Screenshot_from_2024-06-16_15-01-23.png
Attachment 12: Screenshot_from_2024-06-16_15-01-46.png
Screenshot_from_2024-06-16_15-01-46.png
Attachment 13: Screenshot_from_2024-06-16_15-02-12.png
Screenshot_from_2024-06-16_15-02-12.png
Attachment 14: Screenshot_from_2024-06-16_15-03-05.png
Screenshot_from_2024-06-16_15-03-05.png
Attachment 15: Screenshot_from_2024-06-16_15-06-14.png
Screenshot_from_2024-06-16_15-06-14.png
Attachment 16: Screenshot_from_2024-06-16_15-07-06.png
Screenshot_from_2024-06-16_15-07-06.png
Attachment 17: Screenshot_from_2024-06-16_15-08-02.png
Screenshot_from_2024-06-16_15-08-02.png
Attachment 18: R9_747
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261839349 ( 1667447.6 Hz)
        Other data format:      80651 (     513.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        683 (       4.3 Hz)
                               RESUME:        683 (       4.3 Hz)
                              SYNC100:      32585 (     207.5 Hz)
                              WR48-63:      32585 (     207.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      14115 (      89.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     357749 (    2278.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      157.030 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.474                0.000
  1                0.000                0.000
  2                0.114                0.000
  3                0.086                0.000
  4                0.007                0.000
  5                0.165                0.000
  6                0.000                0.000
  7                6.815                0.000
  8                0.000                0.000
  9                3.068                0.000
 10                0.526                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.247                0.000
 14                0.002                0.000
 15                1.373                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   17385482       9389          0          0         56         56       2291       2291          0       4695          0          0
  1    4359471       1110          0          0          0          0        555        555          0          0          0          0
  2   16890768       4312          0          0         14         14       2142       2142          0          0          0          0
  3   16799432       4226          0          0         17         17       2096       2096          0          0          0          0
  4    5895210       1414          0          0          2          2        705        705          0          0          0          0
  5   15605735       4074          0          0         27         27       2010       2010          0          0          0          0
  6    2976504      10096          0          0          0          0        338        338          0       9420          0          0
  7   48753528      12540          0          0        223        223       6047       6047          0          0          0     357748
  8    5419325       1414          0          0          0          0        707        707          0          0          0          0
  9   41464320      10520          0          0        151        151       5109       5109          0          0          0          0
 10   21531781       5458          0          0         66         66       2663       2663          0          0          0          1
 11    3242993        744          0          0          0          0        372        372          0          0          0          0
 12    3988909        970          0          0          0          0        485        485          0          0          0          0
 13   16192913       4074          0          0         38         38       1999       1999          0          0          0          0
 14    4620379       1122          0          0          1          1        560        560          0          0          0          0
 15   36712599       9188          0          0         88         88       4506       4506          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   63.340s (  505.211 blocks/s,  31.576 Mb/s)
Attachment 19: Screenshot_from_2024-06-16_19-11-54.png
Screenshot_from_2024-06-16_19-11-54.png
Attachment 20: Screenshot_from_2024-06-16_19-12-22.png
Screenshot_from_2024-06-16_19-12-22.png
Attachment 21: Screenshot_from_2024-06-16_19-14-40.png
Screenshot_from_2024-06-16_19-14-40.png
Attachment 22: Screenshot_from_2024-06-16_19-15-10.png
Screenshot_from_2024-06-16_19-15-10.png
Attachment 23: Screenshot_from_2024-06-16_23-34-39.png
Screenshot_from_2024-06-16_23-34-39.png
Attachment 24: Screenshot_from_2024-06-16_23-35-06.png
Screenshot_from_2024-06-16_23-35-06.png
Attachment 25: Screenshot_from_2024-06-16_23-35-37.png
Screenshot_from_2024-06-16_23-35-37.png
Attachment 26: Screenshot_from_2024-06-16_23-36-04.png
Screenshot_from_2024-06-16_23-36-04.png
  664   Sat Jun 15 00:28:01 2024 TDSaturday 15 June
Beam lost due to septum failure c. 18.15 Friday 14 June

Cooling of recirculating water failed c. 19.00 - water temperature increased from the normal 17-18deg C to 25-26 deg C
FEE64 temperatures increased and FEE64s powered off and on-call engineer requested.

Cooling of recirculating water restored and AIDA restarted

NH started background run 23.37 - data file S181/R9

01.21 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3
       8x < 20k, max 238k aida08

      per FEE64 Rate spectra - attachment 4

      2x Check ASIC load

      Merger, TapeServer etc - attachments 5-6

      analysis data file R9_27 - attachment 7
       max deadtime 1.9%, all other FEE64s < 1%
       ADC data rate 951k, HEC data rate 1.6kHz

04.28 DSSSD bias & leakage current OK - attachment 8

      FEE64 temperatures OK - attachment 9

      ADC data item stats OK - attachment 10
       8x < 20k, max 261k aida08

      per FEE64 Rate spectra - attachment 11


08.30 DSSSD bias & leakage current OK - attachments 12-13

      FEE64 temperatures OK - attachment 14

      ADC data item stats OK - attachment 15
       all rates increased - gamma source in use?

      per FEE64 Rate spectra - attachments 16-17
       note aida08 asic #3 hot HEC channel

      per FEE64 1.8.L spectra - attachments 18-20
       aida09 pulser peak width 59 ch FWHM
       decay alphas aida05, aida12, aida02, aida04

      per FEE64 1.8.W spectra - 20us FSR - attachments 21-23

      Merger, TapeServer etc - attachments 25-25

      analysis data file R9_152 - attachment 26
       max deadtime 2.4%, all other FEE64s < 1%
       ADC data rate 1.215M, HEC data rate 2.1kHz


12.22 DSSSD bias & leakage current OK - attachment 27

      FEE64 temperatures OK - attachment 28

      ADC data item stats OK - attachment 29

      per FEE64 Rate spectra - attachment 30



15.27 DSSSD bias & leakage current OK - attachment 31

      FEE64 temperatures OK - attachment 32

      ADC data item stats OK - attachment 33
       all rates increased - 370kBq 152Eu in use

      per FEE64 Rate spectra - attachments 34

      per FEE64 1.8.L spectra - attachments 35-36
       aida09 pulser peak width 59 ch FWHM
       decay alphas aida05, aida12, aida02, aida04

      analysis data file R9_275 - attachment 37
       max deadtime 4.4%, all other FEE64s < 2%
       ADC data rate 1.520M, HEC data rate 2.3kHz

18.52 DSSSD bias & leakage current OK - attachment 38

      FEE64 temperatures OK - attachment 39

      ADC data item stats OK - attachment 40

      per FEE64 Rate spectra - attachment 41


22.50 DSSSD bias & leakage current OK - attachment 42

      FEE64 temperatures OK - attachment 43

      ADC data item stats OK - attachment 44

      per FEE64 Rate spectra - attachment 45
Attachment 1: Screenshot_from_2024-06-15_01-21-06.png
Screenshot_from_2024-06-15_01-21-06.png
Attachment 2: Screenshot_from_2024-06-15_01-21-41.png
Screenshot_from_2024-06-15_01-21-41.png
Attachment 3: Screenshot_from_2024-06-15_01-23-59.png
Screenshot_from_2024-06-15_01-23-59.png
Attachment 4: Screenshot_from_2024-06-15_01-24-56.png
Screenshot_from_2024-06-15_01-24-56.png
Attachment 5: Screenshot_from_2024-06-15_01-27-07.png
Screenshot_from_2024-06-15_01-27-07.png
Attachment 6: Screenshot_from_2024-06-15_01-27-34.png
Screenshot_from_2024-06-15_01-27-34.png
Attachment 7: R9_27
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258219194 (  950585.8 Hz)
        Other data format:    3700806 (   13623.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        516 (       1.9 Hz)
                               RESUME:        516 (       1.9 Hz)
                              SYNC100:      33012 (     121.5 Hz)
                              WR48-63:      33012 (     121.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3633750 (   13377.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     437157 (    1609.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      271.642 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.523                0.000
  1                0.000                0.000
  2                0.959                0.000
  3                0.770                0.000
  4                0.002                0.000
  5                0.768                0.000
  6                0.000                0.000
  7                5.278                0.000
  8                0.000                0.000
  9                2.951                0.000
 10                0.780                0.000
 11                0.000                0.000
 12                0.030                0.000
 13                0.033                0.000
 14                0.000                0.000
 15                2.166                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3263639       8999          0          0          7          7        426        426          0       8133          0        129
  1    5814210       1456          0          0          0          0        728        728          0          0          0        128
  2    6330359    1813504          0          0          6          6       1067       1067          0    1811358          0         79
  3   25849187       6768          0          0         50         50       3334       3334          0          0          0        112
  4    2729875        722          0          0          1          1        360        360          0          0          0        127
  5   18600203       4844          0          0         37         37       2385       2385          0          0          0         94
  6    5217587      17698          0          0          0          0        698        698          0      16302          0        124
  7   65848180      17052          0          0        196        196       8330       8330          0          0          0     435453
  8    7897456       2060          0          0          0          0       1030       1030          0          0          0        128
  9   45422925    1809941          0          0        107        107       5885       5885          0    1797957          0         64
 10   15079549       3854          0          0         29         29       1898       1898          0          0          0        111
 11    1722257        384          0          0          0          0        192        192          0          0          0        128
 12    4556679       1158          0          0          1          1        578        578          0          0          0        141
 13    6725939       1702          0          0          2          2        849        849          0          0          0        130
 14    3055032        760          0          0          0          0        380        380          0          0          0        128
 15   40106117       9904          0          0         80         80       4872       4872          0          0          0         81
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   39.733s (  805.368 blocks/s,  50.335 Mb/s)
Attachment 8: Screenshot_from_2024-06-15_04-28-07.png
Screenshot_from_2024-06-15_04-28-07.png
Attachment 9: Screenshot_from_2024-06-15_04-28-40.png
Screenshot_from_2024-06-15_04-28-40.png
Attachment 10: Screenshot_from_2024-06-15_04-30-40.png
Screenshot_from_2024-06-15_04-30-40.png
Attachment 11: Screenshot_from_2024-06-15_04-31-38.png
Screenshot_from_2024-06-15_04-31-38.png
Attachment 12: Screenshot_from_2024-06-15_08-33-07.png
Screenshot_from_2024-06-15_08-33-07.png
Attachment 13: Screenshot_from_2024-06-15_08-30-26.png
Screenshot_from_2024-06-15_08-30-26.png
Attachment 14: Screenshot_from_2024-06-15_08-33-35.png
Screenshot_from_2024-06-15_08-33-35.png
Attachment 15: Screenshot_from_2024-06-15_08-36-46.png
Screenshot_from_2024-06-15_08-36-46.png
Attachment 16: Screenshot_from_2024-06-15_08-35-16.png
Screenshot_from_2024-06-15_08-35-16.png
Attachment 17: Screenshot_from_2024-06-15_08-43-12.png
Screenshot_from_2024-06-15_08-43-12.png
Attachment 18: Screenshot_from_2024-06-15_08-40-42.png
Screenshot_from_2024-06-15_08-40-42.png
Attachment 19: Screenshot_from_2024-06-15_08-39-42.png
Screenshot_from_2024-06-15_08-39-42.png
Attachment 20: Screenshot_from_2024-06-15_08-38-37.png
Screenshot_from_2024-06-15_08-38-37.png
Attachment 21: Screenshot_from_2024-06-15_08-46-26.png
Screenshot_from_2024-06-15_08-46-26.png
Attachment 22: Screenshot_from_2024-06-15_08-45-35.png
Screenshot_from_2024-06-15_08-45-35.png
Attachment 23: Screenshot_from_2024-06-15_08-44-50.png
Screenshot_from_2024-06-15_08-44-50.png
Attachment 24: Screenshot_from_2024-06-15_08-48-10.png
Screenshot_from_2024-06-15_08-48-10.png
Attachment 25: Screenshot_from_2024-06-15_08-47-40.png
Screenshot_from_2024-06-15_08-47-40.png
Attachment 26: R9_152
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259017680 ( 1215243.3 Hz)
        Other data format:    2902320 (   13616.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        515 (       2.4 Hz)
                               RESUME:        515 (       2.4 Hz)
                              SYNC100:      32794 (     153.9 Hz)
                              WR48-63:      32794 (     153.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2835702 (   13304.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     441097 (    2069.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      213.141 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.001                0.000
  1                0.000                0.000
  2                0.002                0.000
  3                0.140                0.000
  4                0.000                0.000
  5                0.047                0.000
  6                0.000                0.000
  7                5.152                0.000
  8                0.000                0.000
  9                2.136                0.000
 10                0.186                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.032                0.000
 14                0.000                0.000
 15                1.354                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8157992       8515          0          0          4          4       1057       1057          0       6393          0          0
  1    4152307       1016          0          0          0          0        508        508          0          0          0          0
  2   10764259    1418465          0          0          2          2       1556       1556          0    1415349          0          0
  3   20158462       5380          0          0         30         30       2660       2660          0          0          0          0
  4    3884729       1032          0          0          0          0        516        516          0          0          0          0
  5   16603608       4178          0          0         21         21       2068       2068          0          0          0          0
  6    5112596      13950          0          0          0          0        582        582          0      12786          0          0
  7   59646516      15586          0          0        217        217       7576       7576          0          0          0     441097
  8    6438302       1678          0          0          0          0        839        839          0          0          0          0
  9   45360093    1412938          0          0        126        126       5756       5756          0    1401174          0          0
 10   18847436       4734          0          0         36         36       2331       2331          0          0          0          0
 11    2364016        648          0          0          0          0        324        324          0          0          0          0
 12    1869299        460          0          0          0          0        230        230          0          0          0          0
 13   11268040       2624          0          0          8          8       1304       1304          0          0          0          0
 14    3774625        934          0          0          0          0        467        467          0          0          0          0
 15   40615400      10182          0          0         71         71       5020       5020          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.906s (  763.609 blocks/s,  47.726 Mb/s)
Attachment 27: Screenshot_from_2024-06-15_12-22-54.png
Screenshot_from_2024-06-15_12-22-54.png
Attachment 28: Screenshot_from_2024-06-15_12-23-19.png
Screenshot_from_2024-06-15_12-23-19.png
Attachment 29: Screenshot_from_2024-06-15_12-25-35.png
Screenshot_from_2024-06-15_12-25-35.png
Attachment 30: Screenshot_from_2024-06-15_12-26-47.png
Screenshot_from_2024-06-15_12-26-47.png
Attachment 31: Screenshot_from_2024-06-15_15-26-54.png
Screenshot_from_2024-06-15_15-26-54.png
Attachment 32: Screenshot_from_2024-06-15_15-27-27.png
Screenshot_from_2024-06-15_15-27-27.png
Attachment 33: Screenshot_from_2024-06-15_15-27-48.png
Screenshot_from_2024-06-15_15-27-48.png
Attachment 34: Screenshot_from_2024-06-15_15-28-25.png
Screenshot_from_2024-06-15_15-28-25.png
Attachment 35: Screenshot_from_2024-06-15_15-31-08.png
Screenshot_from_2024-06-15_15-31-08.png
Attachment 36: Screenshot_from_2024-06-15_15-32-10.png
Screenshot_from_2024-06-15_15-32-10.png
Attachment 37: R9_275
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259129992 ( 1519853.9 Hz)
        Other data format:    2790008 (   16364.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        665 (       3.9 Hz)
                               RESUME:        665 (       3.9 Hz)
                              SYNC100:      32636 (     191.4 Hz)
                              WR48-63:      32636 (     191.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2723406 (   15973.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     395024 (    2316.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      170.497 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.249                0.000
  1                0.000                0.000
  2                0.356                0.000
  3                0.136                0.000
  4                0.337                0.000
  5                0.141                0.000
  6                0.019                0.000
  7                7.476                0.000
  8                0.165                0.000
  9                3.059                0.000
 10                0.581                0.000
 11                0.000                0.000
 12                0.320                0.000
 13                0.705                0.000
 14                0.045                0.000
 15                1.921                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   12819722       8373          0          0         35         35       1597       1597          0       5109          0          0
  1    4488999       1128          0          0          0          0        564        564          0          0          0          0
  2   13809452    1368881          0          0         15         15       1975       1975          0    1364901          0          0
  3   17302926       4370          0          0         25         25       2160       2160          0          0          0          0
  4    4688002       1104          0          0         11         11        541        541          0          0          0          0
  5   15844696       4176          0          0         32         32       2056       2056          0          0          0          0
  6    5051819      11648          0          0          1          1        708        708          0      10230          0          0
  7   52726699      13824          0          0        233        233       6679       6679          0          0          0     395024
  8    5506218       1384          0          0          2          2        690        690          0          0          0          0
  9   42499600    1354256          0          0        142        142       5403       5403          0    1343166          0          0
 10   20647689       5006          0          0         43         43       2460       2460          0          0          0          0
 11    2780744        604          0          0          0          0        302        302          0          0          0          0
 12    3628914        944          0          0          9          9        463        463          0          0          0          0
 13   14531072       3696          0          0         25         25       1823       1823          0          0          0          0
 14    4089821       1130          0          0          3          3        562        562          0          0          0          0
 15   38713619       9484          0          0         89         89       4653       4653          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   49.219s (  650.159 blocks/s,  40.635 Mb/s)
Attachment 38: Screenshot_from_2024-06-15_18-52-04.png
Screenshot_from_2024-06-15_18-52-04.png
Attachment 39: Screenshot_from_2024-06-15_18-52-35.png
Screenshot_from_2024-06-15_18-52-35.png
Attachment 40: Screenshot_from_2024-06-15_18-53-05.png
Screenshot_from_2024-06-15_18-53-05.png
Attachment 41: Screenshot_from_2024-06-15_18-53-49.png
Screenshot_from_2024-06-15_18-53-49.png
Attachment 42: Screenshot_from_2024-06-15_22-50-22.png
Screenshot_from_2024-06-15_22-50-22.png
Attachment 43: Screenshot_from_2024-06-15_22-50-03.png
Screenshot_from_2024-06-15_22-50-03.png
Attachment 44: Screenshot_from_2024-06-15_22-50-49.png
Screenshot_from_2024-06-15_22-50-49.png
Attachment 45: Screenshot_from_2024-06-15_22-51-18.png
Screenshot_from_2024-06-15_22-51-18.png
  663   Fri Jun 14 15:00:47 2024 Norah , Muneerah 16:00-20:00 Friday 14 June

Everything appears to be running smoothly. Attachments 1-4 were taken around 16:00

DSSSD Bias & Leakage Current - Check: OK (see attachment 1)
FEE64 Temperature - Check: OK (see attachment 2)
ADC Data Item Stats and Spectrum Browser - Check: OK (see attachments 3 and 4)

Another set of screenshots was taken around 18:00, and everything looks good.

DSSSD bias & leakage current  ok - attachment # 5-6
FEE64 temperatures  ok - attachment # 7
ADC data item stats - attachment # 8
Merger ok - Attachement #9
Tape service - attachement # 10
ucesb - attachment #11

At 19:17, AIDA03 temperatures gave no response (see attachment 12), but after two minutes, they returned to normal.

I noted that the  bplse in ucesb has no data (see attachment 13). It may need to change the disc thresholds again
 it does not affect AIDA

At 19:30 most aida temperature are high and I spoke to people on site via zoom one of them she is working with AIDA so she was able to help. She called Nick and it is found that there is a problem in the cooling system so she stopped AIDA

At 22 the cooling system fixed and Nic and the other person on site restarted AIDA

 

 

 

 

 

Attachment 1: Screenshot_from_2024-06-14_16-06-09.png
Screenshot_from_2024-06-14_16-06-09.png
Attachment 2: Screenshot_from_2024-06-14_16-04-58.png
Screenshot_from_2024-06-14_16-04-58.png
Attachment 3: Screenshot_from_2024-06-14_16-04-23.png
Screenshot_from_2024-06-14_16-04-23.png
Attachment 4: Screenshot_from_2024-06-14_16-05-33.png
Screenshot_from_2024-06-14_16-05-33.png
Attachment 5: Screenshot_from_2024-06-14_18-02-36.png
Screenshot_from_2024-06-14_18-02-36.png
Attachment 6: Screenshot_from_2024-06-14_18-02-08.png
Screenshot_from_2024-06-14_18-02-08.png
Attachment 7: Screenshot_from_2024-06-14_18-04-40.png
Screenshot_from_2024-06-14_18-04-40.png
Attachment 8: Screenshot_from_2024-06-14_18-05-44.png
Screenshot_from_2024-06-14_18-05-44.png
Attachment 9: Screenshot_from_2024-06-14_18-08-50.png
Screenshot_from_2024-06-14_18-08-50.png
Attachment 10: Screenshot_from_2024-06-14_18-11-04.png
Screenshot_from_2024-06-14_18-11-04.png
Attachment 11: Screenshot_from_2024-06-14_18-17-15.png
Screenshot_from_2024-06-14_18-17-15.png
Attachment 12: Screenshot_from_2024-06-14_19-23-32.png
Screenshot_from_2024-06-14_19-23-32.png
Attachment 13: Screenshot_from_2024-06-14_19-35-11.png
Screenshot_from_2024-06-14_19-35-11.png
  662   Fri Jun 14 06:22:39 2024 TD08:00-16:00 Friday 14 June
07.23 All histograms & stats zero'd

      DAQ continues data file S181/R8_35

      analysis data file R8_35 - attachment 1
      max deadtime 4.8% (aida08), all other FEE64s < 3%
      LEC data rate 1.677M HEC data rate 3.7k

08.46 DSSSD bias & leakage current OK - attachments 2-3

      FEE64 temps OK - attachment 4

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 6

      per p+n FEE64 1.8.L spectra - attachment 8
       aida09 pulser peak width 65 ch FWHM

      per FEE64 1.8.H spectra - attachment 7 & 10

      per FEE64 1.8.W spectra - 20us FSR - attachment 9 & 11

      Merger, TapeServer etc - attachments 12-13
       data file S181/R8_72


10.51 DSSSD bias & leakage current OK - attachments 14

      FEE64 temps OK - attachment 15

      ADC data item stats - attachment 16

      per FEE64 Rate spectra - attachment 17

10.54 analysis data file R8_122 - attachment 18
       max deadtime 7.8% (aida08), all other FEE64s < 3%
       ADC data rate 1.88M, HEC data rate 4.7k


14.24 DSSSD bias & leakage current OK - attachments 18

      FEE64 temps OK - attachment 19

      ADC data item stats - attachment 20

      per FEE64 Rate spectra - attachment 21

      aida02 no data

14.28 merger restarted and DAQ running OK 


14.38 analysis data file R8_213 - attachment 22
       max deadtime 6.6% (aida08), all other FEE64s < 3.5%
       ADC data rate 1.935M, HEC data rate 3.6k
Attachment 1: R8_35
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257896837 ( 1677094.0 Hz)
        Other data format:    4023165 (   26162.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        943 (       6.1 Hz)
                               RESUME:        943 (       6.1 Hz)
                              SYNC100:      32573 (     211.8 Hz)
                              WR48-63:      32573 (     211.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3956133 (   25726.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     571720 (    3717.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      153.776 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.108                0.000
  1                0.164                0.000
  2                0.270                0.000
  3                0.595                0.000
  4                0.126                0.000
  5                1.001                0.000
  6                0.058                0.000
  7                7.436                0.000
  8                0.069                0.000
  9                3.830                0.000
 10                2.186                0.000
 11                0.125                0.000
 12                0.069                0.000
 13                0.757                0.000
 14                0.154                0.000
 15                2.216                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   12981434       7806          0          0         18         18       1578       1578          0       4614          0      11424
  1    8323345       2124          0          0         12         12       1050       1050          0          0          0      25906
  2   13584435    1382069          0          0         11         11       1829       1829          0    1378389          0      15197
  3   18274735       4850          0          0         51         51       2374       2374          0          0          0      20257
  4    5283424     405074          0          0          6          6        727        727          0     403608          0       6463
  5   17252022       4608          0          0         99         99       2205       2205          0          0          0      23562
  6    5456624     322759          0          0          2          2        750        750          0     321255          0      15002
  7   48918628      12422          0          0        216        216       5995       5995          0          0          0     358328
  8    6232233       1470          0          0          2          2        733        733          0          0          0      17364
  9   40119718    1354831          0          0        148        148       5161       5161          0    1344213          0      16391
 10   18973819       5022          0          0        199        199       2312       2312          0          0          0       9722
 11    3858542     505136          0          0          4          4        537        537          0     504054          0      15811
 12    4193350        998          0          0          1          1        498        498          0          0          0       4061
 13   13770630       3644          0          0         62         62       1760       1760          0          0          0      14623
 14    4850419       1244          0          0          2          2        620        620          0          0          0      10415
 15   35823479       9108          0          0        110        110       4444       4444          0          0          0       7194
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.436s ( 1017.956 blocks/s,  63.622 Mb/s)
Attachment 2: Screenshot_from_2024-06-14_08-46-40.png
Screenshot_from_2024-06-14_08-46-40.png
Attachment 3: Screenshot_from_2024-06-14_08-47-06.png
Screenshot_from_2024-06-14_08-47-06.png
Attachment 4: Screenshot_from_2024-06-14_08-49-23.png
Screenshot_from_2024-06-14_08-49-23.png
Attachment 5: Screenshot_from_2024-06-14_08-49-56.png
Screenshot_from_2024-06-14_08-49-56.png
Attachment 6: Screenshot_from_2024-06-14_08-50-17.png
Screenshot_from_2024-06-14_08-50-17.png
Attachment 7: Screenshot_from_2024-06-14_08-53-11.png
Screenshot_from_2024-06-14_08-53-11.png
Attachment 8: Screenshot_from_2024-06-14_08-51-44.png
Screenshot_from_2024-06-14_08-51-44.png
Attachment 9: Screenshot_from_2024-06-14_08-54-47.png
Screenshot_from_2024-06-14_08-54-47.png
Attachment 10: Screenshot_from_2024-06-14_08-53-58.png
Screenshot_from_2024-06-14_08-53-58.png
Attachment 11: Screenshot_from_2024-06-14_08-55-50.png
Screenshot_from_2024-06-14_08-55-50.png
Attachment 12: Screenshot_from_2024-06-14_08-56-36.png
Screenshot_from_2024-06-14_08-56-36.png
Attachment 13: Screenshot_from_2024-06-14_08-56-13.png
Screenshot_from_2024-06-14_08-56-13.png
Attachment 14: Screenshot_from_2024-06-14_10-51-34.png
Screenshot_from_2024-06-14_10-51-34.png
Attachment 15: Screenshot_from_2024-06-14_10-53-46.png
Screenshot_from_2024-06-14_10-53-46.png
Attachment 16: Screenshot_from_2024-06-14_10-54-27.png
Screenshot_from_2024-06-14_10-54-27.png
Attachment 17: Screenshot_from_2024-06-14_10-55-24.png
Screenshot_from_2024-06-14_10-55-24.png
Attachment 18: R8_122
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257243299 ( 1885175.7 Hz)
        Other data format:    4676701 (   34272.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1054 (       7.7 Hz)
                               RESUME:       1054 (       7.7 Hz)
                              SYNC100:      32509 (     238.2 Hz)
                              WR48-63:      32509 (     238.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4609575 (   33780.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     643720 (    4717.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      136.456 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.291                0.000
  1                0.331                0.000
  2                0.072                0.000
  3                0.583                0.000
  4                0.004                0.000
  5                1.153                0.000
  6                0.002                0.000
  7               10.599                0.000
  8                0.009                0.000
  9                3.557                0.000
 10                2.108                0.000
 11                0.007                0.000
 12                0.000                0.000
 13                0.583                0.000
 14                0.004                0.000
 15                1.925                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13787962       7660          0          0         31         31       1756       1756          0       4086          0      17199
  1   11491912       2978          0          0         24         24       1465       1465          0          0          0      38699
  2   14079661    1398532          0          0         10         10       1939       1939          0    1394634          0      23119
  3   19048179       4880          0          0         43         43       2397       2397          0          0          0      31229
  4    5716791     616800          0          0          1          1        782        782          0     615234          0      10346
  5   18363851       4878          0          0        113        113       2326       2326          0          0          0      35150
  6    5742131     471711          0          0          1          1        755        755          0     470199          0      22729
  7   44929108      11550          0          0        247        247       5528       5528          0          0          0     318806
  8    6563442       1628          0          0          4          4        810        810          0          0          0      26254
  9   37095189    1366938          0          0        146        146       4783       4783          0    1357080          0      24684
 10   18370423       4942          0          0        241        241       2230       2230          0          0          0      15154
 11    4582984     769694          0          0          2          2        674        674          0     768342          0      24358
 12    4619008       1036          0          0          0          0        518        518          0          0          0       6521
 13   14138411       3650          0          0         78         78       1747       1747          0          0          0      22204
 14    5321169       1274          0          0          2          2        635        635          0          0          0      16080
 15   33393078       8550          0          0        111        111       4164       4164          0          0          0      11188
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.508s ( 1015.621 blocks/s,  63.476 Mb/s)
Attachment 19: Screenshot_from_2024-06-14_14-23-54.png
Screenshot_from_2024-06-14_14-23-54.png
Attachment 20: Screenshot_from_2024-06-14_14-24-24.png
Screenshot_from_2024-06-14_14-24-24.png
Attachment 21: Screenshot_from_2024-06-14_14-25-10.png
Screenshot_from_2024-06-14_14-25-10.png
Attachment 22: Screenshot_from_2024-06-14_14-25-38.png
Screenshot_from_2024-06-14_14-25-38.png
Attachment 23: R8_213
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258517055 ( 1934722.8 Hz)
        Other data format:    3402945 (   25467.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1143 (       8.6 Hz)
                               RESUME:       1142 (       8.5 Hz)
                              SYNC100:      32498 (     243.2 Hz)
                              WR48-63:      32498 (     243.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3335664 (   24963.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     475836 (    3561.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      133.620 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.550                0.000
  1                0.000                0.000
  2                0.143                0.000
  3                1.051                0.000
  4                0.044                0.000
  5                1.439                0.000
  6                0.003                0.000
  7                8.857                0.000
  8                0.012                0.000
  9                4.685                0.000
 10                3.463                0.000
 11                0.020                0.000
 12                0.000                0.000
 13                0.869                0.000
 14                0.008                0.000
 15                2.432                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   18250814       8751          0          0         55         55       2330       2330          0       3981          0       9142
  1          0          0          0          0          0          0          0          0          0          0          0          0
  2   17501975    1182466          0          0         15         15       2393       2393          0    1177650          0      12677
  3   17523014       4524          0          0         49         49       2213       2213          0          0          0      16471
  4    6950058     338741          0          0          8          8        923        923          0     336879          0       5558
  5   16698357       4354          0          0        104        104       2073       2073          0          0          0      18504
  6    5487299     262839          0          0          1          1        716        716          0     261405          0      12626
  7   43452557      11094          0          0        210        210       5337       5337          0          0          0     321021
  8    5704212       1552          0          0          1          1        775        775          0          0          0      14649
  9   38744177    1147432          0          0        173        172       4922       4922          0    1137243          0      13757
 10   21009531       5724          0          0        299        299       2563       2563          0          0          0       7991
 11    4841879     419852          0          0          4          4        669        669          0     418506          0      13290
 12    4136803       1032          0          0          0          0        516        516          0          0          0       3489
 13   16727954       4328          0          0        114        114       2050       2050          0          0          0      12047
 14    5878222       1412          0          0          1          1        705        705          0          0          0       8774
 15   35610203       8844          0          0        109        109       4313       4313          0          0          0       5840
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.188s (  994.175 blocks/s,  62.136 Mb/s)
  661   Thu Jun 13 23:04:01 2024 Dan Judson, TD0.00-8.00 14/6/24

0.00 checks - all looks ok

Voltage/currents - attachment 1

Rates attachment 2 + 3

Temps - attachment 4

Merger - attachment 5

ucesb - attachment 6

Grafana - attachment 7

 

1.30 -aida1 in merger Links, aida02 in rate spectra, has stopped - resetting DAQ.

Datafile R7_961 being written when stopped

1.40 - seems ok

 

1.50 checks - seems ok

Voltage/currents - attachment 8

Rates attachment 9 + 10

Temps - attachment 11

Merger - attachment 12

ucesb - attachment 13

Grafana - attachment 14

 

3.30 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1010 being written when stopped

3.38 - seems ok

 

3.40 checks - all looks ok

Voltage/currents - attachment 15

Rates attachment 16 + 17

Temps - attachment 18

Merger - attachment 19

ucesb - attachment 20

Grafana - attachment 21

 

Zeroed spectra - attachments 22-28

 

 

5.39 -aida1 in merger Links, aida02 in rate spectra, has stopped again - resetting DAQ.

Datafile R7_1062 being written when stopped

AIDA out of the main data

Could not get it to restart folowing the notes on the elog - SOAP errors as shown in attachment 29. called Tom who''s logged in 

6.10 - seems to be working again. Not sure what happened. Tom thinks multiple restarts fixed it

Started data file R8

AIDA back in the main data

 

FRS shifters report a problem with the AIDA white rabbit time stamp - attachment 30

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  17D8C4ED 8F46CF7B , WR/10=>   2627A17C18714BF, Readout Time =>   2627A17C2A74000
aida02 : White Rabbit=>  17D8C4ED A0A1B623 , WR/10=>   2627A17C3435F03, Readout Time =>   2627A17C4410000
aida03 : White Rabbit=>  17D8C4ED AFDACE68 , WR/10=>   2627A17C4C914A4, Readout Time =>   2627A17C5D3C000
aida04 : White Rabbit=>  17D8C4ED C0516FC2 , WR/10=>   2627A17C66E8B2D, Readout Time =>   2627A17C7AC4000
aida05 : White Rabbit=>  17D8C4ED D3B93DAB , WR/10=>   2627A17C85F52F7, Readout Time =>   2627A17C9AEC000
aida06 : White Rabbit=>  17D8C4ED E8C9B243 , WR/10=>   2627A17CA7A91D3, Readout Time =>   2627A17CB704000
aida07 : White Rabbit=>  17D8C4ED FFE475FD , WR/10=>   2627A17CCCA0BCC, Readout Time =>   2627A17CDD94000
aida08 : White Rabbit=>  17D8C4EE 103232A6 , WR/10=>   2627A17CE6B6B77, Readout Time =>   2627A17CF7BC000
aida09 : White Rabbit=>  17D8C4EE 20A1F612 , WR/10=>   2627A17D0103235, Readout Time =>   2627A17D112C000
aida10 : White Rabbit=>  17D8C4EE 307D8FE7 , WR/10=>   2627A17D1A627FD, Readout Time =>   2627A17D2958000
aida11 : White Rabbit=>  17D8C4EE 4402829D , WR/10=>   2627A17D399D9DC, Readout Time =>   2627A17D4570000
aida12 : White Rabbit=>  17D8C4EE 6288B04D , WR/10=>   2627A17D6A744D4, Readout Time =>   2627A17D7D94000
aida13 : White Rabbit=>  17D8C4EE 751E1306 , WR/10=>   2627A17D88301E7, Readout Time =>   2627A17D9C34000
aida14 : White Rabbit=>  17D8C4EE 894E8806 , WR/10=>   2627A17DA87DA67, Readout Time =>   2627A17DBFF4000
aida15 : White Rabbit=>  17D8C4EE 9EEEB997 , WR/10=>   2627A17DCB178F5, Readout Time =>   2627A17DDC18000
aida16 : White Rabbit=>  17D8C4EE AF0BC923 , WR/10=>   2627A17DE4DFA83, Readout Time =>   2627A17DF5C8000

Tom thinks everything appears ok with AIDA

Have converted WR timestamp to data/time - see attachment 40

At the ~second timescale AIDA WR timestamp looks OK, i.e. no gross errors. Will need to check correlations with other detector sub-systems to confirm WR on the ~us time timescale.. 

However, ~10us offset reported by DESPEC online crew is correct (AIDA timestamps data at a later point in the signal processing cycle than other detector sub-systems). Have suggested online crew contact Nic or Calum to check what the AIDA WR timediff spectra should look like.

 

 

7.00 checks - all looks ok

Voltage/currents - attachment 31

Rates attachment 32 + 33

Temps - attachment 36

Merger - attachment 37

ucesb - attachment 39

Grafana - attachment 38

 

Handed over to Tom

 

Attachment 1: Screenshot_from_2024-06-13_23-58-44.png
Screenshot_from_2024-06-13_23-58-44.png
Attachment 2: Screenshot_from_2024-06-13_23-59-34.png
Screenshot_from_2024-06-13_23-59-34.png
Attachment 3: Screenshot_from_2024-06-14_00-00-09.png
Screenshot_from_2024-06-14_00-00-09.png
Attachment 4: Screenshot_from_2024-06-14_00-00-43.png
Screenshot_from_2024-06-14_00-00-43.png
Attachment 5: Screenshot_from_2024-06-14_00-01-36.png
Screenshot_from_2024-06-14_00-01-36.png
Attachment 6: Screenshot_from_2024-06-14_00-02-45.png
Screenshot_from_2024-06-14_00-02-45.png
Attachment 7: Screenshot_from_2024-06-14_00-03-26.png
Screenshot_from_2024-06-14_00-03-26.png
Attachment 8: Screenshot_from_2024-06-14_01-47-38.png
Screenshot_from_2024-06-14_01-47-38.png
Attachment 9: Screenshot_from_2024-06-14_01-48-12.png
Screenshot_from_2024-06-14_01-48-12.png
Attachment 10: Screenshot_from_2024-06-14_01-48-45.png
Screenshot_from_2024-06-14_01-48-45.png
Attachment 11: Screenshot_from_2024-06-14_01-49-16.png
Screenshot_from_2024-06-14_01-49-16.png
Attachment 12: Screenshot_from_2024-06-14_01-49-48.png
Screenshot_from_2024-06-14_01-49-48.png
Attachment 13: Screenshot_from_2024-06-14_01-50-46.png
Screenshot_from_2024-06-14_01-50-46.png
Attachment 14: Screenshot_from_2024-06-14_01-51-18.png
Screenshot_from_2024-06-14_01-51-18.png
Attachment 15: Screenshot_from_2024-06-14_03-40-33.png
Screenshot_from_2024-06-14_03-40-33.png
Attachment 16: Screenshot_from_2024-06-14_03-40-56.png
Screenshot_from_2024-06-14_03-40-56.png
Attachment 17: Screenshot_from_2024-06-14_03-41-35.png
Screenshot_from_2024-06-14_03-41-35.png
Attachment 18: Screenshot_from_2024-06-14_03-42-15.png
Screenshot_from_2024-06-14_03-42-15.png
Attachment 19: Screenshot_from_2024-06-14_03-42-42.png
Screenshot_from_2024-06-14_03-42-42.png
Attachment 20: Screenshot_from_2024-06-14_03-43-39.png
Screenshot_from_2024-06-14_03-43-39.png
Attachment 21: Screenshot_from_2024-06-14_03-44-11.png
Screenshot_from_2024-06-14_03-44-11.png
Attachment 22: Screenshot_from_2024-06-14_03-53-08.png
Screenshot_from_2024-06-14_03-53-08.png
Attachment 23: Screenshot_from_2024-06-14_03-53-51.png
Screenshot_from_2024-06-14_03-53-51.png
Attachment 24: Screenshot_from_2024-06-14_03-54-36.png
Screenshot_from_2024-06-14_03-54-36.png
Attachment 25: Screenshot_from_2024-06-14_03-55-19.png
Screenshot_from_2024-06-14_03-55-19.png
Attachment 26: Screenshot_from_2024-06-14_03-55-50.png
Screenshot_from_2024-06-14_03-55-50.png
Attachment 27: Screenshot_from_2024-06-14_03-56-56.png
Screenshot_from_2024-06-14_03-56-56.png
Attachment 28: Screenshot_from_2024-06-14_03-58-10.png
Screenshot_from_2024-06-14_03-58-10.png
Attachment 29: Screenshot_from_2024-06-14_05-52-47.png
Screenshot_from_2024-06-14_05-52-47.png
Attachment 30: Screenshot_2024-06-14_at_05.31.21.png
Screenshot_2024-06-14_at_05.31.21.png
Attachment 31: Screenshot_from_2024-06-14_07-02-14.png
Screenshot_from_2024-06-14_07-02-14.png
Attachment 32: Screenshot_from_2024-06-14_07-00-53.png
Screenshot_from_2024-06-14_07-00-53.png
Attachment 33: Screenshot_from_2024-06-14_07-01-20.png
Screenshot_from_2024-06-14_07-01-20.png
Attachment 34: Screenshot_from_2024-06-14_07-01-55.png
Screenshot_from_2024-06-14_07-01-55.png
Attachment 35: Screenshot_from_2024-06-14_07-02-47.png
Screenshot_from_2024-06-14_07-02-47.png
Attachment 36: Screenshot_from_2024-06-14_07-01-55.png
Screenshot_from_2024-06-14_07-01-55.png
Attachment 37: Screenshot_from_2024-06-14_07-02-47.png
Screenshot_from_2024-06-14_07-02-47.png
Attachment 38: Screenshot_from_2024-06-14_07-03-23.png
Screenshot_from_2024-06-14_07-03-23.png
Attachment 39: Screenshot_from_2024-06-14_07-04-01.png
Screenshot_from_2024-06-14_07-04-01.png
Attachment 40: Capture.PNG
Capture.PNG
  660   Thu Jun 13 21:06:44 2024 Robert Page16:00 to 00:00 shift on 13/6/2024

Elog went down around the end of the last shift, so final screenshots were not uploaded at that point.

However, AIDA appears to have contuned running smoothly in the interim (at least until 22:00).

 

Attachments 1 - 6 were taken around 16:00, 7 - 13 at around 18:00 , 14 - 18 at around 21:00.

 

Another set of sceenshots taken at around 23:00 - attachments 19 - 26.

 

 

Attachment 1: Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Attachment 2: Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Attachment 5: Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
Attachment 6: Screenshot_from_2024-06-13_16-04-29.png
Screenshot_from_2024-06-13_16-04-29.png
Attachment 7: Screenshot_from_2024-06-13_17-34-30.png
Screenshot_from_2024-06-13_17-34-30.png
Attachment 8: Screenshot_from_2024-06-13_18-02-57.png
Screenshot_from_2024-06-13_18-02-57.png
Attachment 9: Screenshot_2024-06-13_at_18-01-10_ucesb.png
Screenshot_2024-06-13_at_18-01-10_ucesb.png
Attachment 10: Screenshot_2024-06-13_at_18-05-05_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_18-05-05_Temperature_and_status_scan_aidas-gsi.png
Attachment 11: Screenshot_2024-06-13_at_18-05-39_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_18-05-39_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2024-06-13_at_18-06-06_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_18-06-06_Statistics_aidas-gsi.png
Attachment 13: Screenshot_2024-06-13_at_18-20-26_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_18-20-26_AIDA_-_Grafana.png
Attachment 14: Screenshot_2024-06-13_at_20-55-49_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_20-55-49_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2024-06-13_at_20-56-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_20-56-12_Spectrum_Browser_aidas-gsi.png
Attachment 16: Screenshot_2024-06-13_at_20-56-34_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_20-56-34_Temperature_and_status_scan_aidas-gsi.png
Attachment 17: Screenshot_2024-06-13_at_20-57-39_ucesb.png
Screenshot_2024-06-13_at_20-57-39_ucesb.png
Attachment 18: Screenshot_from_2024-06-13_20-55-04.png
Screenshot_from_2024-06-13_20-55-04.png
Attachment 19: Screenshot_from_2024-06-13_22-52-59.png
Screenshot_from_2024-06-13_22-52-59.png
Attachment 20: Screenshot_2024-06-13_at_22-53-37_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_22-53-37_Statistics_aidas-gsi.png
Attachment 21: Screenshot_2024-06-13_at_22-54-01_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-06-13_at_22-54-01_Spectrum_Browser_aidas-gsi.png
Attachment 22: Screenshot_2024-06-13_at_22-54-24_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_22-54-24_Temperature_and_status_scan_aidas-gsi.png
Attachment 23: Screenshot_2024-06-13_at_22-55-49_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_22-55-49_NewMerger_Control_aidas-gsi.png
Attachment 24: Screenshot_2024-06-13_at_22-56-12_ucesb.png
Screenshot_2024-06-13_at_22-56-12_ucesb.png
Attachment 25: Screenshot_2024-06-13_at_22-56-26_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_22-56-26_AIDA_-_Grafana.png
Attachment 26: Screenshot_2024-06-13_at_22-55-18_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_22-55-18_Tape_Service_(Expert)_aidas-gsi.png
  659   Thu Jun 13 06:49:54 2024 Magda Satrazani08:00 - 16:00 Thursday 13 June 2024

Took over the shift from Marc.

08:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

10:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12

12:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 13-14
FEE64 temperatures  ok - attachment # 15
ADC data item stats - attachment # 16
Merger ok - Attachement # 17
Tape service - attachement # 18

 

 

12.34 analysis data file S181/R7_610 - attachment 19

max deadtime 8.7% (aida08), all other FEE64s < 2.7%

LEC data rate 2.000M, HEC data rate 4.7k

 

12.52 per p+n FEE64 1.8.L spectra - attachment 20

 aida09 pulser peak width 63 ch FWHM

per FEE64 1.8.H spectra - attachments 21-22

per FEE64 1.8.W spectra - 20us FSR - attachments 23-24

 

13.00 all histograms zero'd

 

14:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 19-20
FEE64 temperatures  ok - attachment # 21
ADC data item stats - attachment # 22
Merger ok - Attachement # 23
Tape service - attachement # 24

 

13.01 Beam OFF for S4 access

ADC data item stats - attachment 25

per FEE64 Rate spectra - attachment 26

14.12 analysis data file R7_654 - attachment 27

max deadtime 3.9% (aida08), all other FEE64s < 3%

LEC data rate 1.817M, HEC data rate 2.5k

14.00 data file R7_649 - attachment 28

max deadtime 4.5% (aida08), all other FEE64s < 3%

LEC data rate 1.756M, HEC data rate 2.5k

 

 

16:00 FULL CHECK
DSSSD bias & leakage current  ok - attachment # 25-26
FEE64 temperatures  ok - attachment # 27
ADC data item stats - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30

Attachment 1: Screenshot_2024-06-13_at_07-54-16_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_07-54-16_AIDA_-_Grafana.png
Attachment 2: Screenshot_from_2024-06-13_07-52-18.png
Screenshot_from_2024-06-13_07-52-18.png
Attachment 3: Screenshot_2024-06-13_at_07-56-25_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_07-56-25_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2024-06-13_at_07-58-19_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_07-58-19_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2024-06-13_at_08-02-24_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_08-02-24_NewMerger_Control_aidas-gsi.png
Attachment 6: Screenshot_2024-06-13_at_08-03-42_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_08-03-42_Tape_Service_(Expert)_aidas-gsi.png
Attachment 7: Screenshot_2024-06-13_at_10-03-46_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_10-03-46_AIDA_-_Grafana.png
Attachment 8: Screenshot_from_2024-06-13_10-05-11.png
Screenshot_from_2024-06-13_10-05-11.png
Attachment 9: Screenshot_2024-06-13_at_10-07-46_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_10-07-46_Temperature_and_status_scan_aidas-gsi.png
Attachment 10: Screenshot_2024-06-13_at_10-08-20_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_10-08-20_Statistics_aidas-gsi.png
Attachment 11: Screenshot_2024-06-13_at_10-10-56_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_10-10-56_NewMerger_Control_aidas-gsi.png
Attachment 12: Screenshot_2024-06-13_at_10-11-14_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_10-11-14_Tape_Service_(Expert)_aidas-gsi.png
Attachment 13: Screenshot_2024-06-13_at_12-03-03_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_12-03-03_AIDA_-_Grafana.png
Attachment 14: Screenshot_from_2024-06-13_12-03-51.png
Screenshot_from_2024-06-13_12-03-51.png
Attachment 15: Screenshot_2024-06-13_at_12-04-46_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_12-04-46_Temperature_and_status_scan_aidas-gsi.png
Attachment 16: Screenshot_2024-06-13_at_12-05-04_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_12-05-04_Statistics_aidas-gsi.png
Attachment 17: Screenshot_2024-06-13_at_12-06-12_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_12-06-12_NewMerger_Control_aidas-gsi.png
Attachment 18: Screenshot_2024-06-13_at_12-06-30_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_12-06-30_Tape_Service_(Expert)_aidas-gsi.png
Attachment 19: R7_610
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257594733 ( 2000279.2 Hz)
        Other data format:    4325267 (   33586.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1104 (       8.6 Hz)
                               RESUME:       1104 (       8.6 Hz)
                              SYNC100:      32479 (     252.2 Hz)
                              WR48-63:      32479 (     252.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4258101 (   33065.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     602185 (    4676.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      128.779 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.378                0.000
  1                0.764                0.000
  2                0.139                0.000
  3                0.656                0.000
  4                0.011                0.000
  5                1.499                0.000
  6                0.000                0.000
  7               11.233                0.000
  8                0.013                0.000
  9                3.452                0.000
 10                2.528                0.000
 11                0.011                0.000
 12                0.000                0.000
 13                0.520                0.000
 14                0.005                0.000
 15                2.249                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15133196       7799          0          0         47         47       1922       1922          0       3861          0      15842
  1   11745456       3032          0          0         27         27       1489       1489          0          0          0      35232
  2   14852161    1309647          0          0         18         18       1989       1989          0    1305633          0      20871
  3   18562801       4858          0          0         50         50       2379       2379          0          0          0      27762
  4    6102528     560384          0          0          2          2        773        773          0     558834          0       9246
  5   18133649       4756          0          0        124        124       2254       2254          0          0          0      32497
  6    5708332     425318          0          0          0          0        727        727          0     423864          0      20696
  7   42953157      11164          0          0        244        244       5338       5338          0          0          0     306073
  8    6253278       1506          0          0          2          2        751        751          0          0          0      24189
  9   36712642    1280462          0          0        132        132       4801       4801          0    1270596          0      22769
 10   18971051       5070          0          0        273        273       2262       2262          0          0          0      13720
 11    4672661     696545          0          0          2          2        614        614          0     695313          0      22243
 12    4500765       1148          0          0          1          1        573        573          0          0          0       5817
 13   14826894       3760          0          0         66         66       1814       1814          0          0          0      20573
 14    5503214       1378          0          0          1          1        688        688          0          0          0      14645
 15   32962948       8440          0          0        115        115       4105       4105          0          0          0      10010
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   44.625s (  717.087 blocks/s,  44.818 Mb/s)
Attachment 20: Screenshot_from_2024-06-13_12-52-29.png
Screenshot_from_2024-06-13_12-52-29.png
Attachment 21: Screenshot_from_2024-06-13_12-54-14.png
Screenshot_from_2024-06-13_12-54-14.png
Attachment 22: Screenshot_from_2024-06-13_12-53-37.png
Screenshot_from_2024-06-13_12-53-37.png
Attachment 23: Screenshot_from_2024-06-13_12-55-33.png
Screenshot_from_2024-06-13_12-55-33.png
Attachment 24: Screenshot_from_2024-06-13_12-54-56.png
Screenshot_from_2024-06-13_12-54-56.png
Attachment 25: Screenshot_2024-06-13_at_14-03-30_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_14-03-30_AIDA_-_Grafana.png
Attachment 26: Screenshot_from_2024-06-13_14-03-07.png
Screenshot_from_2024-06-13_14-03-07.png
Attachment 27: Screenshot_2024-06-13_at_14-05-01_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_14-05-01_Temperature_and_status_scan_aidas-gsi.png
Attachment 28: Screenshot_2024-06-13_at_14-05-24_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_14-05-24_Statistics_aidas-gsi.png
Attachment 29: Screenshot_2024-06-13_at_14-06-51_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_14-06-51_NewMerger_Control_aidas-gsi.png
Attachment 30: Screenshot_2024-06-13_at_14-07-11_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_14-07-11_Tape_Service_(Expert)_aidas-gsi.png
Attachment 31: Screenshot_from_2024-06-13_14-17-01.png
Screenshot_from_2024-06-13_14-17-01.png
Attachment 32: Screenshot_from_2024-06-13_14-19-05.png
Screenshot_from_2024-06-13_14-19-05.png
Attachment 33: R7_654
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259951011 ( 1816478.8 Hz)
        Other data format:    1968989 (   13758.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        987 (       6.9 Hz)
                               RESUME:        987 (       6.9 Hz)
                              SYNC100:      32533 (     227.3 Hz)
                              WR48-63:      32533 (     227.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1901949 (   13290.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     354246 (    2475.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      143.107 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.509                0.000
  1                0.000                0.000
  2                0.189                0.000
  3                0.238                0.000
  4                0.004                0.000
  5                0.314                0.000
  6                0.000                0.000
  7                5.616                0.000
  8                0.000                0.000
  9                4.141                0.000
 10                3.021                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.437                0.000
 14                0.000                0.000
 15                2.204                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   17180431       8836          0          0         56         56       2220       2220          0       4284          0          0
  1    3455015        914          0          0          0          0        457        457          0          0          0          0
  2   16605423     962341          0          0         24         24       2144       2144          0     958005          0          0
  3   14977123       3970          0          0         39         39       1946       1946          0          0          0          0
  4    6343544       1554          0          0          1          1        776        776          0          0          0          0
  5   14745967       3758          0          0         68         68       1811       1811          0          0          0          0
  6    4885840       9840          0          0          0          0        627        627          0       8586          0          0
  7   46653553      11994          0          0        167        167       5830       5830          0          0          0     354246
  8    5022167       1302          0          0          0          0        651        651          0          0          0          0
  9   41378629     941868          0          0        154        154       5243       5243          0     931074          0          0
 10   21774767       5944          0          0        304        304       2668       2668          0          0          0          0
 11    3829761       1030          0          0          0          0        515        515          0          0          0          0
 12    3406542        774          0          0          0          0        387        387          0          0          0          0
 13   16951477       4402          0          0         63         63       2138       2138          0          0          0          0
 14    5065059       1154          0          0          0          0        577        577          0          0          0          0
 15   37675713       9308          0          0        111        111       4543       4543          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.719s (  767.041 blocks/s,  47.940 Mb/s)
Attachment 34: R7_649
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259542903 ( 1756063.7 Hz)
        Other data format:    2377097 (   16083.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        995 (       6.7 Hz)
                               RESUME:        994 (       6.7 Hz)
                              SYNC100:      32551 (     220.2 Hz)
                              WR48-63:      32551 (     220.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2310006 (   15629.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     373724 (    2528.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      147.798 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.337                0.000
  1                0.072                0.000
  2                0.087                0.000
  3                0.196                0.000
  4                0.011                0.000
  5                0.401                0.000
  6                0.000                0.000
  7                6.652                0.000
  8                0.000                0.000
  9                3.901                0.000
 10                2.506                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.447                0.000
 14                0.000                0.000
 15                2.451                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   16241246       8589          0          0         45         45       2034       2034          0       4431          0       1142
  1    4109651       1104          0          0          4          4        548        548          0          0          0       2470
  2   15921172    1109604          0          0         15         15       2046       2046          0    1105482          0       1509
  3   15655639       3862          0          0         30         30       1901       1901          0          0          0       1938
  4    5992464      40675          0          0          2          2        792        792          0      39087          0        732
  5   15116945       3940          0          0         57         57       1913       1913          0          0          0       2309
  6    4949702      39054          0          0          0          0        642        642          0      37770          0       1486
  7   47384271      12428          0          0        197        197       6017       6017          0          0          0     352995
  8    5162177       1276          0          0          0          0        638        638          0          0          0       1588
  9   41669335    1085810          0          0        164        163       5344       5344          0    1074795          0       1446
 10   21487589       5832          0          0        286        286       2630       2630          0          0          0        957
 11    3666376      49337          0          0          1          1        447        447          0      48441          0       1546
 12    3498670        892          0          0          0          0        446        446          0          0          0        448
 13   16139721       3982          0          0         64         64       1927       1927          0          0          0       1446
 14    4880733       1226          0          0          0          0        613        613          0          0          0       1032
 15   37667212       9486          0          0        130        130       4613       4613          0          0          0        680
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   46.664s (  685.753 blocks/s,  42.860 Mb/s)
Attachment 35: Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Screenshot_2024-06-13_at_16-02-52_AIDA_-_Grafana.png
Attachment 36: Screenshot_from_2024-06-13_16-04-29.png
Screenshot_from_2024-06-13_16-04-29.png
Attachment 37: Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 38: Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Screenshot_2024-06-13_at_16-05-28_Statistics_aidas-gsi.png
Attachment 39: Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-18_NewMerger_Control_aidas-gsi.png
Attachment 40: Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2024-06-13_at_16-09-41_Tape_Service_(Expert)_aidas-gsi.png
  658   Wed Jun 12 23:00:16 2024 Marc00:00-08:00 Thursday 13 June

Starting a new night shift - All good

1am full check:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

2:30am: AIDA02 FEE stopped working. DESPEC main DAQ was alerted and remobe AIDA from timestitcher while I was restarting AIDA DAQ.

2:45am: AIDA02  FEE is back and AIDA has been added to the timestitcher again in the main DAQ.

The AIDA runs at the time of the crash to look at are R7_348, 349.

All spectra were zeroed at this time and checked.

4:00 am full check:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - attachement # 11
Tape service - attachement # 12
Spectra - attachement # 13
ucesb - attachement # 14

6:00am full check:
DSSSD bias & leakage current  ok - attachment # 15-16
FEE64 temperatures  ok - attachment # 17
ADC data item stats - attachment # 18
Merger ok - attachement # 19
Tape service - attachement # 20
Spectra - attachement # 21
ucesb - attachement # 22

6:30 the nearline/offline analysis team reported possible noise issue in DSSD#2 and asked if the threshold were changed at any point. I can't find any mention of changes in the elog. 
I've not done a setup of the all DAQ when AIDA02 fee crashes earlier this morning. I just did a setup of the merger. Thus I don't think the threshold have changed when I stopeed and restarted the DAQ.
I can see that currently slow comparator threshod are set to 0xa for most of the DSSD2 fees except for AIDA06 which is set to 0xf. I'm not sure this is the thershold we would need to increase. Something to check during the next shift.

In the UCESB window I can see that the number of decay in DSSD2 is double the number of decay in DSSD1 while the number of implant is similar. Looking at other shift it looks that this was alsready the case in prvious shifts.

 

07.06 analysis of data file S181/R7_457 - attachment 23

 max deadtime 5.8% aida08, all other FEE64s < 1.4%

LEC data rate 1.788M, HEC data rate 4.3k

 

Attachment 1: Screenshot_from_2024-06-13_01-04-30.png
Screenshot_from_2024-06-13_01-04-30.png
Attachment 2: Screenshot_from_2024-06-13_01-06-23.png
Screenshot_from_2024-06-13_01-06-23.png
Attachment 3: Screenshot_from_2024-06-13_01-07-02.png
Screenshot_from_2024-06-13_01-07-02.png
Attachment 4: Screenshot_from_2024-06-13_01-07-36.png
Screenshot_from_2024-06-13_01-07-36.png
Attachment 5: Screenshot_from_2024-06-13_01-08-39.png
Screenshot_from_2024-06-13_01-08-39.png
Attachment 6: Screenshot_from_2024-06-13_01-08-02.png
Screenshot_from_2024-06-13_01-08-02.png
Attachment 7: Screenshot_from_2024-06-13_04-41-04.png
Screenshot_from_2024-06-13_04-41-04.png
Attachment 8: Screenshot_from_2024-06-13_04-41-38.png
Screenshot_from_2024-06-13_04-41-38.png
Attachment 9: Screenshot_from_2024-06-13_04-42-19.png
Screenshot_from_2024-06-13_04-42-19.png
Attachment 10: Screenshot_from_2024-06-13_04-42-45.png
Screenshot_from_2024-06-13_04-42-45.png
Attachment 11: Screenshot_from_2024-06-13_04-46-50.png
Screenshot_from_2024-06-13_04-46-50.png
Attachment 12: Screenshot_from_2024-06-13_04-47-09.png
Screenshot_from_2024-06-13_04-47-09.png
Attachment 13: Screenshot_from_2024-06-13_04-45-02.png
Screenshot_from_2024-06-13_04-45-02.png
Attachment 14: Screenshot_from_2024-06-13_04-48-07.png
Screenshot_from_2024-06-13_04-48-07.png
Attachment 15: Screenshot_from_2024-06-13_06-01-55.png
Screenshot_from_2024-06-13_06-01-55.png
Attachment 16: Screenshot_from_2024-06-13_06-02-13.png
Screenshot_from_2024-06-13_06-02-13.png
Attachment 17: Screenshot_from_2024-06-13_06-03-06.png
Screenshot_from_2024-06-13_06-03-06.png
Attachment 18: Screenshot_from_2024-06-13_06-02-41.png
Screenshot_from_2024-06-13_06-02-41.png
Attachment 19: Screenshot_from_2024-06-13_06-04-13.png
Screenshot_from_2024-06-13_06-04-13.png
Attachment 20: Screenshot_from_2024-06-13_06-04-37.png
Screenshot_from_2024-06-13_06-04-37.png
Attachment 21: Screenshot_from_2024-06-13_06-03-37.png
Screenshot_from_2024-06-13_06-03-37.png
Attachment 22: Screenshot_from_2024-06-13_06-05-30.png
Screenshot_from_2024-06-13_06-05-30.png
Attachment 23: R7_457
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257532265 ( 1788132.4 Hz)
        Other data format:    4387735 (   30465.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1032 (       7.2 Hz)
                               RESUME:       1032 (       7.2 Hz)
                              SYNC100:      32537 (     225.9 Hz)
                              WR48-63:      32537 (     225.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4320597 (   29999.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     613796 (    4261.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      144.023 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.236                0.000
  1                0.214                0.000
  2                0.096                0.000
  3                0.448                0.000
  4                0.012                0.000
  5                0.887                0.000
  6                0.003                0.000
  7                8.383                0.000
  8                0.002                0.000
  9                3.230                0.000
 10                1.587                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.540                0.000
 14                0.000                0.000
 15                1.978                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13645957       7863          0          0         38         38       1738       1738          0       4311          0      14717
  1    9668775       2474          0          0         22         22       1215       1215          0          0          0      34051
  2   13971257    1386114          0          0         14         14       1900       1900          0    1382286          0      19760
  3   18385419       4726          0          0         51         51       2312       2312          0          0          0      26170
  4    5575749     526316          0          0          3          3        775        775          0     524760          0       8695
  5   17895997       4760          0          0        110        110       2270       2270          0          0          0      31206
  6    5682061     408056          0          0          1          1        783        783          0     406488          0      19551
  7   46828757      11780          0          0        204        204       5686       5686          0          0          0     333264
  8    6455085       1532          0          0          1          1        765        765          0          0          0      22949
  9   38428621    1359707          0          0        141        141       4975       4975          0    1349475          0      21660
 10   18781575       5148          0          0        247        247       2327       2327          0          0          0      13075
 11    4229184     654561          0          0          1          1        641        641          0     653277          0      20787
 12    4432128       1092          0          0          0          0        546        546          0          0          0       5445
 13   14014813       3450          0          0         86         86       1639       1639          0          0          0      19095
 14    5156679       1244          0          0          0          0        622        622          0          0          0      13849
 15   34380208       8912          0          0        113        113       4343       4343          0          0          0       9522
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   52.717s (  607.017 blocks/s,  37.939 Mb/s)
  657   Wed Jun 12 15:41:07 2024 Betool Alayed16:00-00:00 Wed 12 Jun 24

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6

7pm full checks:
DSSSD bias & leakage current  ok - attachment # 7-8
FEE64 temperatures  ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12
ucesb - attachment # 13

9pm full checks:
DSSSD bias & leakage current ok - attachment# 14-15
FEE64 temperatures ok - attachment # 16
ADC data item stats - attachment # 17
Merger ok - Attachement # 18
Tape service - attachement # 19
ucesb - attachment # 20

11pm full checks:
DSSSD bias & leakage current ok - attachment# 21-22
FEE64 temperatures ok - attachment # 23
ADC data item stats - attachment # 24
Merger ok - Attachement # 25
Tape service - attachement # 26
ucesb - attachment # 27

Attachment 1: Screenshot_from_2024-06-12_17-00-31.png
Screenshot_from_2024-06-12_17-00-31.png
Attachment 2: Screenshot_from_2024-06-12_17-01-10.png
Screenshot_from_2024-06-12_17-01-10.png
Attachment 3: Screenshot_from_2024-06-12_17-02-29.png
Screenshot_from_2024-06-12_17-02-29.png
Attachment 4: Screenshot_from_2024-06-12_17-03-14.png
Screenshot_from_2024-06-12_17-03-14.png
Attachment 5: Screenshot_from_2024-06-12_17-31-47.png
Screenshot_from_2024-06-12_17-31-47.png
Attachment 6: Screenshot_from_2024-06-12_17-31-13.png
Screenshot_from_2024-06-12_17-31-13.png
Attachment 7: Screenshot_from_2024-06-12_19-03-51.png
Screenshot_from_2024-06-12_19-03-51.png
Attachment 8: Screenshot_from_2024-06-12_19-04-39.png
Screenshot_from_2024-06-12_19-04-39.png
Attachment 9: Screenshot_from_2024-06-12_19-05-34.png
Screenshot_from_2024-06-12_19-05-34.png
Attachment 10: Screenshot_from_2024-06-12_19-06-05.png
Screenshot_from_2024-06-12_19-06-05.png
Attachment 11: Screenshot_from_2024-06-12_19-07-15.png
Screenshot_from_2024-06-12_19-07-15.png
Attachment 12: Screenshot_from_2024-06-12_19-08-02.png
Screenshot_from_2024-06-12_19-08-02.png
Attachment 13: Screenshot_from_2024-06-12_19-12-15.png
Screenshot_from_2024-06-12_19-12-15.png
Attachment 14: Screenshot_from_2024-06-12_21-01-54.png
Screenshot_from_2024-06-12_21-01-54.png
Attachment 15: Screenshot_from_2024-06-12_21-03-08.png
Screenshot_from_2024-06-12_21-03-08.png
Attachment 16: Screenshot_from_2024-06-12_21-03-52.png
Screenshot_from_2024-06-12_21-03-52.png
Attachment 17: Screenshot_from_2024-06-12_21-04-48.png
Screenshot_from_2024-06-12_21-04-48.png
Attachment 18: Screenshot_from_2024-06-12_21-06-37.png
Screenshot_from_2024-06-12_21-06-37.png
Attachment 19: Screenshot_from_2024-06-12_21-07-16.png
Screenshot_from_2024-06-12_21-07-16.png
Attachment 20: Screenshot_from_2024-06-12_21-08-36.png
Screenshot_from_2024-06-12_21-08-36.png
Attachment 21: Screenshot_from_2024-06-12_22-58-59.png
Screenshot_from_2024-06-12_22-58-59.png
Attachment 22: Screenshot_from_2024-06-12_22-59-43.png
Screenshot_from_2024-06-12_22-59-43.png
Attachment 23: Screenshot_from_2024-06-12_23-01-09.png
Screenshot_from_2024-06-12_23-01-09.png
Attachment 24: Screenshot_from_2024-06-12_23-02-14.png
Screenshot_from_2024-06-12_23-02-14.png
Attachment 25: Screenshot_from_2024-06-12_23-03-09.png
Screenshot_from_2024-06-12_23-03-09.png
Attachment 26: Screenshot_from_2024-06-12_23-03-48.png
Screenshot_from_2024-06-12_23-03-48.png
Attachment 27: Screenshot_from_2024-06-12_23-05-26.png
Screenshot_from_2024-06-12_23-05-26.png
  656   Wed Jun 12 11:06:55 2024 PPMid-shift checks, 12:00

All seems normal.

Screenshots attached.

Attachment 1: Screenshot_from_2024-06-12_12-07-32.png
Screenshot_from_2024-06-12_12-07-32.png
Attachment 2: Screenshot_from_2024-06-12_12-07-59.png
Screenshot_from_2024-06-12_12-07-59.png
Attachment 3: Screenshot_from_2024-06-12_12-08-24.png
Screenshot_from_2024-06-12_12-08-24.png
Attachment 4: Screenshot_from_2024-06-12_12-09-09.png
Screenshot_from_2024-06-12_12-09-09.png
Attachment 5: Screenshot_from_2024-06-12_12-10-02.png
Screenshot_from_2024-06-12_12-10-02.png
Attachment 6: Screenshot_from_2024-06-12_12-10-30.png
Screenshot_from_2024-06-12_12-10-30.png
Attachment 7: Screenshot_from_2024-06-12_12-10-46.png
Screenshot_from_2024-06-12_12-10-46.png
Attachment 8: Screenshot_from_2024-06-12_12-11-41.png
Screenshot_from_2024-06-12_12-11-41.png
  655   Wed Jun 12 06:17:23 2024 TD08:00-16:00 Wednesday 12 June
07:10 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      Merger, TapeServer etc - attachment 5

      per p+n FEE64 1.8.L spectra - attachment 6
       aida09 pulser peak width 68 ch FWHM 
07.13  peak area 25253 total area 27548 in 221mins => 1.9-2.1Hz => deadtime < 5% or less which is consistent with data file analysis below

      analysis data file S181/R6_274 - attachment 
       max FEE64 deadtime 24% aida08
       HEC data rate 4.5kHz, LEC data rate 2.284M


14.00 DAQ crashed by accident
      FEE64s power cycled 
      DAQ restarted
      data file S181/R7

      DSSSD bias & leakage current OK - attachments 8-9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11

      WR timestamps OK - attachment 12

      All system wide checks *except* as reported attachments 13-15

      per FEE64 Rate spectra - attachment 16

      per p+n FEE64 1.8.L spectra - attachment 17
       aida09 pulser peak width 66 ch FWHM

      per FEE64 1.8.H spectra - attachments 18-19

      per FEE64 1.8.W spectra - 20us FSR - attachment 20-21

      Merger, TapeServer etc - attachments 22-23
       data file S181/R7_16

      analysis data file S181/R7_16
       max deadtime 7.3% aida08, all other FEE64s < 2%
       LEC data rate 1.877M HEC data rate 4.7k
Attachment 1: Screenshot_from_2024-06-12_07-10-38.png
Screenshot_from_2024-06-12_07-10-38.png
Attachment 2: Screenshot_from_2024-06-12_07-13-55.png
Screenshot_from_2024-06-12_07-13-55.png
Attachment 3: Screenshot_from_2024-06-12_07-14-23.png
Screenshot_from_2024-06-12_07-14-23.png
Attachment 4: Screenshot_from_2024-06-12_07-15-39.png
Screenshot_from_2024-06-12_07-15-39.png
Attachment 5: Screenshot_from_2024-06-12_07-16-37.png
Screenshot_from_2024-06-12_07-16-37.png
Attachment 6: Screenshot_from_2024-06-12_07-13-22.png
Screenshot_from_2024-06-12_07-13-22.png
Attachment 7: R6_506
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258515762 ( 2284450.0 Hz)
        Other data format:    3404238 (   30082.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1713 (      15.1 Hz)
                               RESUME:       1713 (      15.1 Hz)
                              SYNC100:      32421 (     286.5 Hz)
                              WR48-63:      32421 (     286.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3335970 (   29479.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     510832 (    4514.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      113.163 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.702                0.000
  1                9.626                0.000
  2                0.301                0.000
  3               21.630                0.000
  4                2.306                0.000
  5               22.211                0.000
  6                0.011                0.000
  7               26.657                0.000
  8                0.007                0.000
  9                3.393                0.000
 10                2.227                0.000
 11                0.257                0.000
 12                1.375                0.000
 13                0.345                0.000
 14                0.006                0.000
 15                1.380                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13756770       7112          0          0         62         62       1805       1805          0       3378          0      31438
  1   16571054       4544          0          0        118        118       2154       2154          0          0          0      40517
  2   12913225    1100195          0          0         22         22       1734       1734          0    1096683          0      15005
  3   22509949       6366          0          0        388        388       2795       2795          0          0          0      25941
  4   11433492     382287          0          0         38         38       1426       1426          0     379359          0      11107
  5   23747115       6330          0          0        227        227       2938       2938          0          0          0      25861
  6    5606649     311005          0          0          2          2        708        708          0     309585          0      13358
  7   37077077       9680          0          0        293        293       4547       4547          0          0          0     194815
  8    5480756       1296          0          0          1          1        647        647          0          0          0      14388
  9   29391154    1072320          0          0        141        141       3828       3828          0    1064382          0      12206
 10   15104540       4112          0          0        210        210       1846       1846          0          0          0       7729
 11    8762807     484875          0          0         23         23       1123       1123          0     482583          0      43011
 12   10240819       2646          0          0         83         83       1240       1240          0          0          0      10084
 13   13149998       3348          0          0         39         39       1635       1635          0          0          0      22309
 14    4835687       1232          0          0          1          1        615        615          0          0          0       8794
 15   27934670       6890          0          0         65         65       3380       3380          0          0          0      34269
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.514s (  876.384 blocks/s,  54.774 Mb/s)
Attachment 8: Screenshot_from_2024-06-12_14-48-17.png
Screenshot_from_2024-06-12_14-48-17.png
Attachment 9: Screenshot_from_2024-06-12_14-48-10.png
Screenshot_from_2024-06-12_14-48-10.png
Attachment 10: Screenshot_from_2024-06-12_14-48-34.png
Screenshot_from_2024-06-12_14-48-34.png
Attachment 11: Screenshot_from_2024-06-12_14-48-44.png
Screenshot_from_2024-06-12_14-48-44.png
Attachment 12: Screenshot_from_2024-06-12_14-50-11.png
Screenshot_from_2024-06-12_14-50-11.png
Attachment 13: Screenshot_from_2024-06-12_14-49-42.png
Screenshot_from_2024-06-12_14-49-42.png
Attachment 14: Screenshot_from_2024-06-12_14-49-38.png
Screenshot_from_2024-06-12_14-49-38.png
Attachment 15: Screenshot_from_2024-06-12_14-49-03.png
Screenshot_from_2024-06-12_14-49-03.png
Attachment 16: Screenshot_from_2024-06-12_14-50-44.png
Screenshot_from_2024-06-12_14-50-44.png
Attachment 17: Screenshot_from_2024-06-12_14-52-04.png
Screenshot_from_2024-06-12_14-52-04.png
Attachment 18: Screenshot_from_2024-06-12_14-53-15.png
Screenshot_from_2024-06-12_14-53-15.png
Attachment 19: Screenshot_from_2024-06-12_14-52-48.png
Screenshot_from_2024-06-12_14-52-48.png
Attachment 20: Screenshot_from_2024-06-12_14-54-44.png
Screenshot_from_2024-06-12_14-54-44.png
Attachment 21: Screenshot_from_2024-06-12_14-54-19.png
Screenshot_from_2024-06-12_14-54-19.png
Attachment 22: Screenshot_from_2024-06-12_14-55-06.png
Screenshot_from_2024-06-12_14-55-06.png
Attachment 23: Screenshot_from_2024-06-12_14-54-57.png
Screenshot_from_2024-06-12_14-54-57.png
Attachment 24: R7_16
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257426271 ( 1876729.6 Hz)
        Other data format:    4493729 (   32760.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        958 (       7.0 Hz)
                               RESUME:        958 (       7.0 Hz)
                              SYNC100:      32511 (     237.0 Hz)
                              WR48-63:      32511 (     237.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4426791 (   32272.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     640590 (    4670.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      137.167 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.216                0.000
  1                0.956                0.000
  2                0.109                0.000
  3                1.065                0.000
  4                0.003                0.000
  5                1.328                0.000
  6                0.000                0.000
  7               10.048                0.000
  8                0.006                0.000
  9                2.496                0.000
 10                1.879                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.358                0.000
 14                0.002                0.000
 15                1.716                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13230081       7571          0          0         32         32       1703       1703          0       4101          0      17508
  1   13090516       3490          0          0         40         40       1705       1705          0          0          0      37780
  2   13947474    1365581          0          0         17         17       1952       1952          0    1361643          0      23114
  3   19918409       5228          0          0         54         54       2560       2560          0          0          0      29816
  4    5368632     580346          0          0          2          2        689        689          0     578964          0       9807
  5   18843779       4956          0          0        116        116       2362       2362          0          0          0      35331
  6    5796835     443912          0          0          1          1        735        735          0     442440          0      22936
  7   45421367      11632          0          0        240        240       5576       5576          0          0          0     316545
  8    6655009       1606          0          0          2          2        801        801          0          0          0      26785
  9   36628182    1346978          0          0        117        117       4723       4723          0    1337298          0      25590
 10   18208001       4832          0          0        188        188       2228       2228          0          0          0      15146
 11    4458699     703685          0          0          0          0        670        670          0     702345          0      24257
 12    4535451       1002          0          0          0          0        501        501          0          0          0       5991
 13   13425648       3478          0          0         54         54       1685       1685          0          0          0      22832
 14    5214701       1310          0          0          1          1        654        654          0          0          0      16212
 15   32683487       8122          0          0         94         94       3967       3967          0          0          0      10940
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   42.707s (  749.291 blocks/s,  46.831 Mb/s)
  654   Tue Jun 11 23:13:44 2024 TD00:00-08:00 Wednesday 11 June
00.08 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 3

      ADC data items stats OK - attachment 2
   
      per FEE64 Rate spectra - attachment 7

      per p+n FEE64 1.8.L spectra - attachments 6
       aida09 pulser peak width 63 ch FWHM 

      Merger, TapeServer etc - attachments 4-5

      analysis data file S181/R6_274 - attachment 8
       max FEE64 deadtime 19% aida08
       HEC data rate 3.8kHz, LEC data rate 2.258M

      per FEE64 1.8.W spectra - 20us FSR - attachments 9-10

03.12 Merger, TapeServer etc - attachments 11-12
       before & after Merger restart

      per FEE64 Rate spectra - attachment 13

      ADC data items stats OK - attachment 14
       aida02 zero rate

      FEE64 temperatures OK - attachment 15

      DSSSD bias & leakage current OK - attachment 16

      analysis data file S181/R6_372 - attachment 17
       max FEE64 deadtime 22% aida08
       HEC data rate 4.4kHz, LEC data rate 2.295M

03.32 all histograms zero'd

03.57 per p+n FEE64 1.8.L spectra - attachment 18
       25 mins = 1500 @ 2Hz = 3000 cf. peak area c. 3000 => deadtime low
Attachment 1: Screenshot_from_2024-06-12_00-08-05.png
Screenshot_from_2024-06-12_00-08-05.png
Attachment 2: Screenshot_from_2024-06-12_00-11-04.png
Screenshot_from_2024-06-12_00-11-04.png
Attachment 3: Screenshot_from_2024-06-12_00-08-40.png
Screenshot_from_2024-06-12_00-08-40.png
Attachment 4: Screenshot_from_2024-06-12_00-13-13.png
Screenshot_from_2024-06-12_00-13-13.png
Attachment 5: Screenshot_from_2024-06-12_00-12-56.png
Screenshot_from_2024-06-12_00-12-56.png
Attachment 6: Screenshot_from_2024-06-12_00-12-08.png
Screenshot_from_2024-06-12_00-12-08.png
Attachment 7: Screenshot_from_2024-06-12_00-15-36.png
Screenshot_from_2024-06-12_00-15-36.png
Attachment 8: R6_274
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258999041 ( 2257638.6 Hz)
        Other data format:    2920959 (   25461.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1511 (      13.2 Hz)
                               RESUME:       1510 (      13.2 Hz)
                              SYNC100:      32427 (     282.7 Hz)
                              WR48-63:      32427 (     282.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2853084 (   24869.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     439283 (    3829.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      114.721 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.601                0.000
  1                7.539                0.000
  2                0.326                0.000
  3               16.402                0.000
  4                0.635                0.000
  5               17.425                0.000
  6                0.097                0.000
  7               21.859                0.000
  8                0.128                0.000
  9                3.673                0.000
 10                3.675                0.000
 11                0.154                0.000
 12                0.891                0.000
 13                0.524                0.000
 14                0.022                0.000
 15                2.309                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15369403       7472          0          0         46         46       1974       1974          0       3432          0      22292
  1   15769502       4294          0          0        112        112       2035       2035          0          0          0      29638
  2   14433962    1024458          0          0         14         14       1879       1879          0    1020672          0      10278
  3   20761922       5804          0          0        295        295       2607       2607          0          0          0      20061
  4    9340460     275709          0          0         19         19       1175       1175          0     273321          0       7943
  5   21823754       5845          0          0        181        180       2742       2742          0          0          0      18704
  6    5422542     230099          0          0          5          5        701        701          0     228687          0       9316
  7   37349907       9452          0          0        244        244       4482       4482          0          0          0     214200
  8    5081034       1202          0          0          1          1        600        600          0          0          0      10181
  9   31593425     999213          0          0        129        129       4047       4047          0     990861          0       8738
 10   17180710       4686          0          0        246        246       2097       2097          0          0          0       5196
 11    7163253     337989          0          0         11         11        928        928          0     336111          0      29914
 12    8343534       2202          0          0         50         50       1051       1051          0          0          0       7185
 13   14807749       3744          0          0         49         49       1823       1823          0          0          0      15937
 14    5467474       1324          0          0          3          3        659        659          0          0          0       5944
 15   29090410       7466          0          0        106        106       3627       3627          0          0          0      23756
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.896s (  867.303 blocks/s,  54.206 Mb/s)
Attachment 9: Screenshot_from_2024-06-12_00-24-11.png
Screenshot_from_2024-06-12_00-24-11.png
Attachment 10: Screenshot_from_2024-06-12_00-23-16.png
Screenshot_from_2024-06-12_00-23-16.png
Attachment 11: Screenshot_from_2024-06-12_03-11-52.png
Screenshot_from_2024-06-12_03-11-52.png
Attachment 12: Screenshot_from_2024-06-12_03-05-47.png
Screenshot_from_2024-06-12_03-05-47.png
Attachment 13: Screenshot_from_2024-06-12_03-04-36.png
Screenshot_from_2024-06-12_03-04-36.png
Attachment 14: Screenshot_from_2024-06-12_03-04-13.png
Screenshot_from_2024-06-12_03-04-13.png
Attachment 15: Screenshot_from_2024-06-12_03-02-04.png
Screenshot_from_2024-06-12_03-02-04.png
Attachment 16: Screenshot_from_2024-06-12_03-01-26.png
Screenshot_from_2024-06-12_03-01-26.png
Attachment 17: R6_372
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260609951 ( 2295050.9 Hz)
        Other data format:    1310049 (   11536.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1765 (      15.5 Hz)
                               RESUME:       1765 (      15.5 Hz)
                              SYNC100:      32423 (     285.5 Hz)
                              WR48-63:      32423 (     285.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1241673 (   10934.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     498232 (    4387.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      113.553 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.716                0.000
  1                0.000                0.000
  2                0.675                0.000
  3               20.755                0.000
  4                8.029                0.000
  5               21.275                0.000
  6                0.025                0.000
  7               25.466                0.000
  8                0.004                0.000
  9                3.222                0.000
 10                3.079                0.000
 11                0.541                0.000
 12                3.417                0.000
 13                0.637                0.000
 14                0.101                0.000
 15                2.523                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   16545027       7593          0          0         61         61       2039       2039          0       3393          0      34807
  1          0          0          0          0          0          0          0          0          0          0          0          0
  2   16018011       4126          0          0         31         31       2032       2032          0          0          0      16400
  3   22563914       6244          0          0        367        367       2755       2755          0          0          0      27855
  4   14584632     380495          0          0        101        101       1904       1904          0     376485          0      10790
  5   23574124       6600          0          0        239        239       3061       3061          0          0          0      28773
  6    6172794     333349          0          0          6          6        797        797          0     331743          0      14869
  7   36993806       9704          0          0        287        287       4565       4565          0          0          0     198994
  8    6095601       1620          0          0          1          1        809        809          0          0          0      16000
  9   30899077       7806          0          0        122        122       3781       3781          0          0          0      13820
 10   16520099       4484          0          0        262        262       1980       1980          0          0          0       8141
 11   10197807     532664          0          0         39         39       1267       1267          0     530052          0      46994
 12   11352337       3042          0          0        119        119       1402       1402          0          0          0      10315
 13   14645959       3712          0          0         38         38       1818       1818          0          0          0      24695
 14    5727639       1460          0          0          7          7        723        723          0          0          0       9258
 15   28719124       7150          0          0         85         85       3490       3490          0          0          0      36521
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   37.054s (  863.611 blocks/s,  53.976 Mb/s)
Attachment 18: Screenshot_from_2024-06-12_03-57-50.png
Screenshot_from_2024-06-12_03-57-50.png
  653   Tue Jun 11 20:20:36 2024 TDADC data rates from 09.00 11.6.24
Per NH

"From ELOG (assuming times accurateish)

10:00 Source in (AIDA removed from DESPEC DAQ)
11:30: Source out (?)
11:45: Background Runs
13:00: bPlast thresholds changed
15:45: Beam Returned"
-
                  ADC data item rate
EOF time  file    LEC        HEC
09.02    R4_538   0.906M     1.7k
10.02      _551   0.927M     1.7k
11.00      _564   0.875M     1.7k
11.19      _568   0.872M     1.7k
12.18    R6_7     1.121M     1.9k
13.03     6_19    1.134M     1.9k
14.02     6_35    1.197M     2.0k
15.00     6_52    1.232M     2.0k
15.10     6_54    1.225M     2.0k
15.36     6_56    1.467M     2.0k
16.00     6_65    1.630M     3.0k
17.01     6_89    1.778M     3.7k
21.43     6_207   1.759M     2.7k
12.6.24
00.09     6_274   2.258M     3.8k
03.12     6_372   2.295M     4.4k
07.13     6_506   2.284M     4.5k
11.12     6_611   1.853M     3.6k
14.55     7_16    1.877M     4.7k
13.6.24
07.01     7_457   1.788M     4.3k
12.34     7_610   2.000M     4.7k
14.00     7_649   1.756M     2.5k
14.12     7_654   1.817M     2.5k
15.33     7_693   1.907M     3.5k
14.6.24
07.23     8_35    1.677M     3.7k
10.54     8_122   1.880M     4.7k
14.38     8_213   1.935M     3.6k
15.6.24
01.25     9_27    0.951k     1.6k  (beam off from c. 18.15 14.6.24)
08.56     9_152   1.215M     2.1k  gamma source in use?
15.27     9_275   1.520M     2.3k  370kBq 152Eu source in use
16.6.24
15.06     9_747   1.667M     2.3k

15.06 analysis data file R9_747 - attachment 18
       max dead time 4.3% aida08, all other FEE64s < 2%
       ADC data rate 1.667M, HEC data rate 2.3k 
  652   Tue Jun 11 18:45:37 2024 PPMid-shift checks, 19:45

All looks good.

Sceenshots attached.

Attachment 1: Screenshot_from_2024-06-11_19-46-41.png
Screenshot_from_2024-06-11_19-46-41.png
Attachment 2: Screenshot_from_2024-06-11_19-47-09.png
Screenshot_from_2024-06-11_19-47-09.png
Attachment 3: Screenshot_from_2024-06-11_19-47-38.png
Screenshot_from_2024-06-11_19-47-38.png
Attachment 4: Screenshot_from_2024-06-11_19-48-12.png
Screenshot_from_2024-06-11_19-48-12.png
Attachment 5: Screenshot_from_2024-06-11_19-48-58.png
Screenshot_from_2024-06-11_19-48-58.png
Attachment 6: Screenshot_from_2024-06-11_19-49-22.png
Screenshot_from_2024-06-11_19-49-22.png
Attachment 7: Screenshot_from_2024-06-11_19-49-43.png
Screenshot_from_2024-06-11_19-49-43.png
Attachment 8: Screenshot_from_2024-06-11_19-50-19.png
Screenshot_from_2024-06-11_19-50-19.png
  651   Tue Jun 11 15:01:07 2024 PP TD16:00-00:00 shift Tuesday 11/06/24
All seems OK

Screenshots attached.

17.33 per FEE64 1.8.W spectra - attachments 8-9
      per p+n FEE64 1.8.L spectra - attachment 10
       aida09 pulser peak width 66 ch FWHM
       cf 
       c. 11:00 today 57 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/650
       cf
       c. 00:10 today 53 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/649

17.48 ADC data item stats - attachment 11
      All histograms zero'd

17.51 Analysis data file S181/R6_96 - attachment 12
      max deadtime 5% (aida08)
      HEC Data items 3.7kHz, LEC data items 1.8MHz
      cf.
      c. 12.29 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/650
      c. 00.09 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/649
Attachment 1: Screenshot_from_2024-06-11_16-06-43.png
Screenshot_from_2024-06-11_16-06-43.png
Attachment 2: Screenshot_from_2024-06-11_16-07-53.png
Screenshot_from_2024-06-11_16-07-53.png
Attachment 3: Screenshot_from_2024-06-11_16-09-03.png
Screenshot_from_2024-06-11_16-09-03.png
Attachment 4: Screenshot_from_2024-06-11_16-12-05.png
Screenshot_from_2024-06-11_16-12-05.png
Attachment 5: Screenshot_from_2024-06-11_16-12-46.png
Screenshot_from_2024-06-11_16-12-46.png
Attachment 6: Screenshot_from_2024-06-11_16-13-40.png
Screenshot_from_2024-06-11_16-13-40.png
Attachment 7: Screenshot_from_2024-06-11_16-14-11.png
Screenshot_from_2024-06-11_16-14-11.png
Attachment 8: Screenshot_from_2024-06-11_16-17-21.png
Screenshot_from_2024-06-11_16-17-21.png
Attachment 9: Screenshot_from_2024-06-11_17-34-26.png
Screenshot_from_2024-06-11_17-34-26.png
Attachment 10: Screenshot_from_2024-06-11_17-33-33.png
Screenshot_from_2024-06-11_17-33-33.png
Attachment 11: Screenshot_from_2024-06-11_17-39-32.png
Screenshot_from_2024-06-11_17-39-32.png
Attachment 12: Screenshot_from_2024-06-11_17-48-43.png
Screenshot_from_2024-06-11_17-48-43.png
Attachment 13: R6_98
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258260025 ( 1785474.1 Hz)
        Other data format:    3659975 (   25303.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        939 (       6.5 Hz)
                               RESUME:        938 (       6.5 Hz)
                              SYNC100:      32538 (     225.0 Hz)
                              WR48-63:      32538 (     225.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3593022 (   24840.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     539819 (    3732.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      144.645 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.468                0.000
  1                0.015                0.000
  2                0.040                0.000
  3                0.872                0.000
  4                0.007                0.000
  5                0.312                0.000
  6                0.000                0.000
  7                7.299                0.000
  8                0.001                0.000
  9                3.731                0.000
 10                2.112                0.000
 11                0.008                0.000
 12                0.000                0.000
 13                0.367                0.000
 14                0.001                0.000
 15                1.636                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15794317       8531          0          0         49         49       2058       2058          0       4317          0      10696
  1    7121332       1752          0          0          3          3        873        873          0          0          0      24880
  2   15560192    1259782          0          0          6          6       2078       2078          0    1255614          0      14645
  3   17271150       4664          0          0        134        134       2198       2198          0          0          0      19004
  4    6664272     367237          0          0          2          2        840        840          0     365553          0       5226
  5   16529211       4220          0          0         48         48       2062       2062          0          0          0      23130
  6    5459932     300460          0          0          0          0        680        680          0     299100          0      14450
  7   45345978      11675          0          0        200        199       5638       5638          0          0          0     336191
  8    6229388       1534          0          0          1          1        766        766          0          0          0      16825
  9   38606423    1232596          0          0        155        155       5043       5043          0    1222200          0      15969
 10   19820101       5140          0          0        195        195       2375       2375          0          0          0       9560
 11    4555885     447544          0          0          2          2        651        651          0     446238          0      14419
 12    4015114        976          0          0          0          0        488        488          0          0          0       3706
 13   15417116       3872          0          0         46         46       1890       1890          0          0          0      14088
 14    5845488       1420          0          0          1          1        709        709          0          0          0      10117
 15   34024126       8572          0          0         97         97       4189       4189          0          0          0       6913
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.520s (  876.243 blocks/s,  54.765 Mb/s)
  650   Tue Jun 11 10:03:48 2024 TD, NH08:00-16:00 Tuesday 10 June
10.57 DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2

      ADC, MBS correlation scaler data item stats - attachments 3-4
       ADC 9x < 20k, max 243k aida08

      ASIC settings aida01 & aida02 - attachments 5-6

      ALL system wide checks OK *except* attachments 7-8

      WR timestamps OK - attachment 9

      BNC PB-5 settings - attachment 10

      per FEE64 Rate spectra - attachment 11

      per p+n FEE64 1.8.L spectra - attachment 12
       aida09 pulser peak width 57 ch FWHM

      per FEE64 1.8.H spectra - attachments 13-14

      per FEE64 1.8.W spectra - 20us FSR - attachments 15-16
       no p+n FEE64 wavefroms - reason unknown

      Merger, TapeServer etc - attachments 17-18

11.25 Lost control of aida-gsi window system 
      All command terminals lost, could not start new command terminals

      aida-gsi restart

      FEE64 power cycle
      DSSSD bias cycle

      Note file manager, calculator apps now working as expected

12.11 DAQ starts 
      data file S181/R6

      ASIC settings 2024Apr19-15.22.49

      BNC PB-5 Pulser
      Amplitude 10.0V
      Attenuation x10
      Fall time 1ms
      Frequency 2Hz
      Polarity + (connected to test + signal daisy chain to all p+n FEE64s)

      Sync ASIC clocks OK
      All system wide checks OK *except* aida02 FPGA/WR decoder - attachments 19-20

      per FEE64 1.8.W spectra - 20us FSR - attachments 21-22

12.29 analysis data file S181/R6_9 - attachment 23
      max dead time c. 2% (aida08)

15:34 For some reason the AIDA MBS wasn't connecting to the time sorter (or a check stream) [NH]
      Try to reset the AIDA MBS 
      Due to confusion the AIDA Merger was rebooted too, while the DAQ was running, this broke all the FEEs -> Powecycled
      Synced ASIC clocks, fixed all waveforms
      Masked all discriminators
      All looks ok -> Sending back to MBS and back in DESPEC DAQ
Attachment 1: Screenshot_from_2024-06-11_10-56-57.png
Screenshot_from_2024-06-11_10-56-57.png
Attachment 2: Screenshot_from_2024-06-11_10-57-13.png
Screenshot_from_2024-06-11_10-57-13.png
Attachment 3: Screenshot_from_2024-06-11_10-57-29.png
Screenshot_from_2024-06-11_10-57-29.png
Attachment 4: Screenshot_from_2024-06-11_10-57-44.png
Screenshot_from_2024-06-11_10-57-44.png
Attachment 5: Screenshot_from_2024-06-11_11-02-06.png
Screenshot_from_2024-06-11_11-02-06.png
Attachment 6: Screenshot_from_2024-06-11_11-02-12.png
Screenshot_from_2024-06-11_11-02-12.png
Attachment 7: Screenshot_from_2024-06-11_11-02-29.png
Screenshot_from_2024-06-11_11-02-29.png
Attachment 8: Screenshot_from_2024-06-11_11-02-33.png
Screenshot_from_2024-06-11_11-02-33.png
Attachment 9: Screenshot_from_2024-06-11_11-02-57.png
Screenshot_from_2024-06-11_11-02-57.png
Attachment 10: Screenshot_from_2024-06-11_11-03-07.png
Screenshot_from_2024-06-11_11-03-07.png
Attachment 11: Screenshot_from_2024-06-11_10-57-57.png
Screenshot_from_2024-06-11_10-57-57.png
Attachment 12: Screenshot_from_2024-06-11_10-59-40.png
Screenshot_from_2024-06-11_10-59-40.png
Attachment 13: Screenshot_from_2024-06-11_11-00-40.png
Screenshot_from_2024-06-11_11-00-40.png
Attachment 14: Screenshot_from_2024-06-11_11-01-11.png
Screenshot_from_2024-06-11_11-01-11.png
Attachment 15: Screenshot_from_2024-06-11_11-01-27.png
Screenshot_from_2024-06-11_11-01-27.png
Attachment 16: Screenshot_from_2024-06-11_11-01-52.png
Screenshot_from_2024-06-11_11-01-52.png
Attachment 17: Screenshot_from_2024-06-11_11-03-24.png
Screenshot_from_2024-06-11_11-03-24.png
Attachment 18: Screenshot_from_2024-06-11_11-03-33.png
Screenshot_from_2024-06-11_11-03-33.png
Attachment 19: Screenshot_from_2024-06-11_12-18-20.png
Screenshot_from_2024-06-11_12-18-20.png
Attachment 20: Screenshot_from_2024-06-11_12-18-15.png
Screenshot_from_2024-06-11_12-18-15.png
Attachment 21: Screenshot_from_2024-06-11_12-23-35.png
Screenshot_from_2024-06-11_12-23-35.png
Attachment 22: Screenshot_from_2024-06-11_12-22-13.png
Screenshot_from_2024-06-11_12-22-13.png
Attachment 23: R6_9
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258839908 ( 1138557.8 Hz)
        Other data format:    3080092 (   13548.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        641 (       2.8 Hz)
                               RESUME:        641 (       2.8 Hz)
                              SYNC100:      32847 (     144.5 Hz)
                              WR48-63:      32847 (     144.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3013116 (   13253.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     428993 (    1887.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      227.340 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.006                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.570                0.000
  4                0.000                0.000
  5                0.059                0.000
  6                0.000                0.000
  7                4.510                0.000
  8                0.000                0.000
  9                2.812                0.000
 10                0.098                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.022                0.000
 14                0.000                0.000
 15                1.041                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8222140       8845          0          0          2          2       1011       1011          0       6819          0          0
  1    3606751        920          0          0          0          0        460        460          0          0          0          0
  2   10679748    1508816          0          0          0          0       1558       1558          0    1505700          0          0
  3   19436638       5222          0          0        115        115       2496       2496          0          0          0          0
  4    3900385        924          0          0          0          0        462        462          0          0          0          0
  5   15462846       3992          0          0         26         26       1970       1970          0          0          0          0
  6    5052319      14939          0          0          0          0        649        649          0      13641          0          0
  7   59921630      15444          0          0        211        211       7511       7511          0          0          0     428993
  8    6654004       1590          0          0          0          0        795        795          0          0          0          0
  9   48245181    1499946          0          0        182        182       6313       6313          0    1486956          0          0
 10   16829911       4196          0          0         17         17       2081       2081          0          0          0          0
 11    2314011        558          0          0          0          0        279        279          0          0          0          0
 12    3820239        906          0          0          0          0        453        453          0          0          0          0
 13   11619760       2934          0          0          7          7       1460       1460          0          0          0          0
 14    3764196        930          0          0          0          0        465        465          0          0          0          0
 15   39310149       9930          0          0         81         81       4884       4884          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.500s ( 1084.746 blocks/s,  67.797 Mb/s)
  649   Mon Jun 10 23:36:14 2024 TD0000-08:00 Tuesday 11 June
00.09 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data items stats OK - attachment 3
   
      per FEE64 Rate spectra - attachment 4

      per p+n FEE64 1.8.L spectra - attachments 5-6
       aida09 pulser peak width 53 ch FWHM 

      per FEE64 1.8.H spectra - attachments 7-8

      per n+n FEE64 1.8.W spectra - 20us FSR - attachment 9
       p+n 1.8.W not running currently

      Merger, TapeServer etc - attachments 10-11

      analysis data file S181/R4_427 - attachment 12
       max FEE64 deadtime 1.7%
       HEC data rate 3.1kHz

00.28 all histograms zero'd


02.39 DAQ continues data file S181/R4_460
      Beam off since 02.08

      DSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data items stats OK - attachment 15
       9x < 20k, max 239k aida08
   
      per FEE64 Rate spectra - attachment 16


07.15 DAQ continues data file S181/R4_517
      Beam off since 02.08
      HKR report c 3-4 h until UNILAC fixed

      DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18

      ADC data items stats OK - attachment 19
       8x < 20k, max 238k aida08
   
      per FEE64 Rate spectra - attachment 20
Attachment 1: Screenshot_from_2024-06-11_00-09-43.png
Screenshot_from_2024-06-11_00-09-43.png
Attachment 2: Screenshot_from_2024-06-11_00-10-36.png
Screenshot_from_2024-06-11_00-10-36.png
Attachment 3: Screenshot_from_2024-06-11_00-11-13.png
Screenshot_from_2024-06-11_00-11-13.png
Attachment 4: Screenshot_from_2024-06-11_00-11-45.png
Screenshot_from_2024-06-11_00-11-45.png
Attachment 5: Screenshot_from_2024-06-11_00-13-25.png
Screenshot_from_2024-06-11_00-13-25.png
Attachment 6: Screenshot_from_2024-06-11_00-16-14.png
Screenshot_from_2024-06-11_00-16-14.png
Attachment 7: Screenshot_from_2024-06-11_00-24-51.png
Screenshot_from_2024-06-11_00-24-51.png
Attachment 8: Screenshot_from_2024-06-11_00-25-44.png
Screenshot_from_2024-06-11_00-25-44.png
Attachment 9: Screenshot_from_2024-06-11_00-27-29.png
Screenshot_from_2024-06-11_00-27-29.png
Attachment 10: Screenshot_from_2024-06-11_00-29-21.png
Screenshot_from_2024-06-11_00-29-21.png
Attachment 11: Screenshot_from_2024-06-11_00-29-41.png
Screenshot_from_2024-06-11_00-29-41.png
Attachment 12: R4_427
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256031206 ( 1090943.7 Hz)
        Other data format:    5888794 (   25092.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        689 (       2.9 Hz)
                               RESUME:        689 (       2.9 Hz)
                              SYNC100:      32874 (     140.1 Hz)
                              WR48-63:      32874 (     140.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5821668 (   24806.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     725584 (    3091.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      234.688 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.082                0.000
  1                0.081                0.000
  2                0.099                0.000
  3                1.580                0.000
  4                0.860                0.000
  5                0.325                0.000
  6                0.000                0.000
  7                4.102                0.000
  8                0.020                0.000
  9                2.330                0.000
 10                0.170                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.981                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3851535       8075          0          0          3          3        517        517          0       7035          0      14311
  1   10547580       2760          0          0          9          9       1371       1371          0          0          0      35298
  2    7097044    2015426          0          0          1          1       1095       1095          0    2013234          0      19431
  3   24895003       6544          0          0        192        192       3080       3080          0          0          0      27887
  4    3219755     585586          0          0          4          4        493        493          0     584592          0      17915
  5   19996310       5104          0          0         46         46       2506       2506          0          0          0      32222
  6    5825374     446331          0          0          0          0        741        741          0     444849          0      18892
  7   61314555      15906          0          0        191        191       7762       7762          0          0          0     429462
  8    8374860       2050          0          0          1          1       1024       1024          0          0          0      21845
  9   43291179    2005473          0          0        168        168       5805       5805          0    1993527          0      21309
 10   13902520       3510          0          0          6          6       1749       1749          0          0          0      12563
 11    2813051     779317          0          0          0          0        443        443          0     778431          0      21093
 12    5093380       1266          0          0          0          0        633        633          0          0          0      11468
 13    7142579       1774          0          0          0          0        887        887          0          0          0      18724
 14    3712134        902          0          0          0          0        451        451          0          0          0      12892
 15   34954347       8770          0          0         68         68       4317       4317          0          0          0      10272
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.627s (  924.132 blocks/s,  57.758 Mb/s)
Attachment 13: Screenshot_from_2024-06-11_02-36-59.png
Screenshot_from_2024-06-11_02-36-59.png
Attachment 14: Screenshot_from_2024-06-11_02-41-01.png
Screenshot_from_2024-06-11_02-41-01.png
Attachment 15: Screenshot_from_2024-06-11_02-41-50.png
Screenshot_from_2024-06-11_02-41-50.png
Attachment 16: Screenshot_from_2024-06-11_02-42-40.png
Screenshot_from_2024-06-11_02-42-40.png
Attachment 17: Screenshot_from_2024-06-11_07-13-28.png
Screenshot_from_2024-06-11_07-13-28.png
Attachment 18: Screenshot_from_2024-06-11_07-15-41.png
Screenshot_from_2024-06-11_07-15-41.png
Attachment 19: Screenshot_from_2024-06-11_07-16-08.png
Screenshot_from_2024-06-11_07-16-08.png
Attachment 20: Screenshot_from_2024-06-11_07-16-35.png
Screenshot_from_2024-06-11_07-16-35.png
  648   Mon Jun 10 15:07:24 2024 Marc16:00-00:00 shift Monday 10/06/24

Evening shift

5pm full checks:
DSSSD bias & leakage current  ok - attachment # 1-2
FEE64 temperatures  ok - attachment # 3
ADC data item stats - attachment # 4
per FEE64 Rate spectra - attachment # 5
Merger ok - Attachement # 6
Tape service - attachement # 7
7pm full checks:
DSSSD bias & leakage current  ok - attachment # 8-9
FEE64 temperatures  ok - attachment # 10
ADC data item stats - attachment # 11
per FEE64 Rate spectra - attachment # 12
Merger ok - Attachement # 13
Tape service - attachement # 14
ucesb - attachment # 15

9pm full checks:
DSSSD bias & leakage current ok - attachment# 16-17
FEE64 temperatures ok - attachment # 18
ADC data item stats - attachment # 19
per FEE64 Rate spectra - attachment # 20
Merger ok - Attachement # 21
Tape service - attachement # 22
ucesb - attachment # 23

9pm full checks:
DSSSD bias & leakage current ok - attachment# 24-25
FEE64 temperatures ok - attachment # 26
ADC data item stats - attachment # 27
per FEE64 Rate spectra - attachment # 28
Merger ok - Attachement # 29
Tape service - attachement # 30
ucesb - attachment # 31

 

Attachment 1: Screenshot_from_2024-06-10_16-54-55.png
Screenshot_from_2024-06-10_16-54-55.png
Attachment 2: Screenshot_from_2024-06-10_16-55-40.png
Screenshot_from_2024-06-10_16-55-40.png
Attachment 3: Screenshot_from_2024-06-10_17-01-34.png
Screenshot_from_2024-06-10_17-01-34.png
Attachment 4: Screenshot_from_2024-06-10_17-02-22.png
Screenshot_from_2024-06-10_17-02-22.png
Attachment 5: Screenshot_from_2024-06-10_17-05-47.png
Screenshot_from_2024-06-10_17-05-47.png
Attachment 6: Screenshot_from_2024-06-10_17-06-52.png
Screenshot_from_2024-06-10_17-06-52.png
Attachment 7: Screenshot_from_2024-06-10_17-13-25.png
Screenshot_from_2024-06-10_17-13-25.png
Attachment 8: Screenshot_from_2024-06-10_18-59-09.png
Screenshot_from_2024-06-10_18-59-09.png
Attachment 9: Screenshot_from_2024-06-10_18-59-42.png
Screenshot_from_2024-06-10_18-59-42.png
Attachment 10: Screenshot_from_2024-06-10_19-02-24.png
Screenshot_from_2024-06-10_19-02-24.png
Attachment 11: Screenshot_from_2024-06-10_19-01-43.png
Screenshot_from_2024-06-10_19-01-43.png
Attachment 12: Screenshot_from_2024-06-10_19-03-42.png
Screenshot_from_2024-06-10_19-03-42.png
Attachment 13: Screenshot_from_2024-06-10_19-04-24.png
Screenshot_from_2024-06-10_19-04-24.png
Attachment 14: Screenshot_from_2024-06-10_19-05-59.png
Screenshot_from_2024-06-10_19-05-59.png
Attachment 15: Screenshot_from_2024-06-10_19-07-37.png
Screenshot_from_2024-06-10_19-07-37.png
Attachment 16: Screenshot_from_2024-06-10_20-56-43.png
Screenshot_from_2024-06-10_20-56-43.png
Attachment 17: Screenshot_from_2024-06-10_20-57-19.png
Screenshot_from_2024-06-10_20-57-19.png
Attachment 18: Screenshot_from_2024-06-10_20-59-07.png
Screenshot_from_2024-06-10_20-59-07.png
Attachment 19: Screenshot_from_2024-06-10_20-59-34.png
Screenshot_from_2024-06-10_20-59-34.png
Attachment 20: Screenshot_from_2024-06-10_21-00-23.png
Screenshot_from_2024-06-10_21-00-23.png
Attachment 21: Screenshot_from_2024-06-10_21-02-53.png
Screenshot_from_2024-06-10_21-02-53.png
Attachment 22: Screenshot_from_2024-06-10_21-02-04.png
Screenshot_from_2024-06-10_21-02-04.png
Attachment 23: Screenshot_from_2024-06-10_21-08-56.png
Screenshot_from_2024-06-10_21-08-56.png
Attachment 24: Screenshot_from_2024-06-10_22-57-04.png
Screenshot_from_2024-06-10_22-57-04.png
Attachment 25: Screenshot_from_2024-06-10_22-57-32.png
Screenshot_from_2024-06-10_22-57-32.png
Attachment 26: Screenshot_from_2024-06-10_22-58-10.png
Screenshot_from_2024-06-10_22-58-10.png
Attachment 27: Screenshot_from_2024-06-10_22-58-48.png
Screenshot_from_2024-06-10_22-58-48.png
Attachment 28: Screenshot_from_2024-06-10_22-59-34.png
Screenshot_from_2024-06-10_22-59-34.png
Attachment 29: Screenshot_from_2024-06-10_23-00-43.png
Screenshot_from_2024-06-10_23-00-43.png
Attachment 30: Screenshot_from_2024-06-10_23-01-18.png
Screenshot_from_2024-06-10_23-01-18.png
Attachment 31: Screenshot_from_2024-06-10_23-03-00.png
Screenshot_from_2024-06-10_23-03-00.png
  647   Mon Jun 10 07:15:48 2024 Dan08.00-16.00 Monday 10th June

8am checks

temps ok - attachment 2

rates ok - attachment 8

voltage ok - attachment 5

merger ok - attachment  6

 

9am checks

temps ok - attachment 10

rates ok - attachment 9

voltage ok - attachment 12

merger ok - attachment  11

 

(Ignore  Grafana plots 13-43 inc - not refreshed)

 

9.16 beam stopped to optimise beam intensity

 

10.00 checks - still no beam

temps ok - attachment 17

rates ok - attachment 15+16

voltage ok - attachment 14

merger ok - attachment  18

 

11.00 checks - still no beam

temps ok - attachment 23

rates ok - attachment 21+22

voltage ok - attachment 20

merger ok - attachment  24

 

12pm checks

temps ok - attachment 28

rates ok - attachment 26+27

voltage ok - attachment 29

merger ok - attachment  30

 

12.45 - beam back

 

temps ok - attachment 34

rates ok - attachment 32+33

voltage ok - attachment 35

merger ok - attachment  36

 

2pm checks

temps ok - attachment 40

rates ok - attachment 38+39

voltage ok - attachment 41

merger ok - attachment  42

 

/tmp/R4_276 added - attachment 44

max deadtime 10.5% (aida08)

n+n FEE64s deadtime 

 

3pm checks

temps ok - attachment 48

rates ok - attachment 46+47

voltage ok - attachment 45+50

merger ok - attachment  49

Attachment 1: Screenshot_from_2024-06-10_08-07-10.png
Screenshot_from_2024-06-10_08-07-10.png
Attachment 2: Screenshot_from_2024-06-10_08-07-54.png
Screenshot_from_2024-06-10_08-07-54.png
Attachment 3: Screenshot_from_2024-06-10_08-11-27.png
Screenshot_from_2024-06-10_08-11-27.png
Attachment 4: Screenshot_from_2024-06-10_08-13-25.png
Screenshot_from_2024-06-10_08-13-25.png
Attachment 5: Screenshot_from_2024-06-10_08-22-19.png
Screenshot_from_2024-06-10_08-22-19.png
Attachment 6: Screenshot_from_2024-06-10_08-28-14.png
Screenshot_from_2024-06-10_08-28-14.png
Attachment 7: Screenshot_from_2024-06-10_08-33-04.png
Screenshot_from_2024-06-10_08-33-04.png
Attachment 8: Screenshot_from_2024-06-10_09-00-04.png
Screenshot_from_2024-06-10_09-00-04.png
Attachment 9: Screenshot_from_2024-06-10_09-00-35.png
Screenshot_from_2024-06-10_09-00-35.png
Attachment 10: Screenshot_from_2024-06-10_09-01-05.png
Screenshot_from_2024-06-10_09-01-05.png
Attachment 11: Screenshot_from_2024-06-10_09-03-04.png
Screenshot_from_2024-06-10_09-03-04.png
Attachment 12: Screenshot_from_2024-06-10_09-03-40.png
Screenshot_from_2024-06-10_09-03-40.png
Attachment 13: Screenshot_from_2024-06-10_09-04-27.png
Screenshot_from_2024-06-10_09-04-27.png
Attachment 14: Screenshot_from_2024-06-10_10-04-59.png
Screenshot_from_2024-06-10_10-04-59.png
Attachment 15: Screenshot_from_2024-06-10_10-05-36.png
Screenshot_from_2024-06-10_10-05-36.png
Attachment 16: Screenshot_from_2024-06-10_10-06-07.png
Screenshot_from_2024-06-10_10-06-07.png
Attachment 17: Screenshot_from_2024-06-10_10-06-40.png
Screenshot_from_2024-06-10_10-06-40.png
Attachment 18: Screenshot_from_2024-06-10_10-07-24.png
Screenshot_from_2024-06-10_10-07-24.png
Attachment 19: Screenshot_from_2024-06-10_10-09-01.png
Screenshot_from_2024-06-10_10-09-01.png
Attachment 20: Screenshot_from_2024-06-10_11-00-43.png
Screenshot_from_2024-06-10_11-00-43.png
Attachment 21: Screenshot_from_2024-06-10_11-01-36.png
Screenshot_from_2024-06-10_11-01-36.png
Attachment 22: Screenshot_from_2024-06-10_11-02-19.png
Screenshot_from_2024-06-10_11-02-19.png
Attachment 23: Screenshot_from_2024-06-10_11-03-06.png
Screenshot_from_2024-06-10_11-03-06.png
Attachment 24: Screenshot_from_2024-06-10_11-03-58.png
Screenshot_from_2024-06-10_11-03-58.png
Attachment 25: Screenshot_from_2024-06-10_11-04-27.png
Screenshot_from_2024-06-10_11-04-27.png
Attachment 26: Screenshot_from_2024-06-10_12-03-07.png
Screenshot_from_2024-06-10_12-03-07.png
Attachment 27: Screenshot_from_2024-06-10_12-03-36.png
Screenshot_from_2024-06-10_12-03-36.png
Attachment 28: Screenshot_from_2024-06-10_12-04-16.png
Screenshot_from_2024-06-10_12-04-16.png
Attachment 29: Screenshot_from_2024-06-10_12-04-48.png
Screenshot_from_2024-06-10_12-04-48.png
Attachment 30: Screenshot_from_2024-06-10_12-05-27.png
Screenshot_from_2024-06-10_12-05-27.png
Attachment 31: Screenshot_from_2024-06-10_12-06-06.png
Screenshot_from_2024-06-10_12-06-06.png
Attachment 32: Screenshot_from_2024-06-10_12-45-12.png
Screenshot_from_2024-06-10_12-45-12.png
Attachment 33: Screenshot_from_2024-06-10_12-45-39.png
Screenshot_from_2024-06-10_12-45-39.png
Attachment 34: Screenshot_from_2024-06-10_12-46-16.png
Screenshot_from_2024-06-10_12-46-16.png
Attachment 35: Screenshot_from_2024-06-10_12-46-53.png
Screenshot_from_2024-06-10_12-46-53.png
Attachment 36: Screenshot_from_2024-06-10_12-50-29.png
Screenshot_from_2024-06-10_12-50-29.png
Attachment 37: Screenshot_from_2024-06-10_12-51-03.png
Screenshot_from_2024-06-10_12-51-03.png
Attachment 38: Screenshot_from_2024-06-10_13-54-46.png
Screenshot_from_2024-06-10_13-54-46.png
Attachment 39: Screenshot_from_2024-06-10_13-55-20.png
Screenshot_from_2024-06-10_13-55-20.png
Attachment 40: Screenshot_from_2024-06-10_13-55-55.png
Screenshot_from_2024-06-10_13-55-55.png
Attachment 41: Screenshot_from_2024-06-10_13-56-15.png
Screenshot_from_2024-06-10_13-56-15.png
Attachment 42: Screenshot_from_2024-06-10_13-57-11.png
Screenshot_from_2024-06-10_13-57-11.png
Attachment 43: Screenshot_from_2024-06-10_13-57-46.png
Screenshot_from_2024-06-10_13-57-46.png
Attachment 44: R4_276
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257167172 ( 1454701.6 Hz)
        Other data format:    4752828 (   26885.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1210 (       6.8 Hz)
                               RESUME:       1214 (       6.9 Hz)
                              SYNC100:      32658 (     184.7 Hz)
                              WR48-63:      32658 (     184.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4685088 (   26501.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     717535 (    4058.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      176.783 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.207                0.000
  1               13.018                0.000
  2                0.307                0.000
  3               15.029                0.000
  4                0.460                0.000
  5               16.031                0.000
  6                0.053                0.000
  7               18.555                0.000
  8                0.064                0.000
  9                2.767                0.000
 10                0.156                0.000
 11                0.013                0.000
 12                0.042                0.000
 13                0.120                0.000
 14                0.011                0.000
 15                1.165                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6985399       7079          0          0         19         19        870        870          0       5301          0      23500
  1   14902829       3959          0          0        133        134       1846       1846          0          0          0      30106
  2    9849800    1542328          0          0         15         15       1400       1400          0    1539498          0      34705
  3   24099351       6585          0          0        273        274       3019       3019          0          0          0      21224
  4    4358270     608906          0          0         22         22        570        570          0     607722          0      20659
  5   21810185       6107          0          0        208        209       2845       2845          0          0          0      26202
  6    6709059     419440          0          0          8          8        900        900          0     417624          0      34989
  7   50592433      13110          0          0        251        251       6304       6304          0          0          0     314546
  8    8465407       2084          0          0          5          5       1037       1037          0          0          0      35796
  9   38024962    1522758          0          0        170        171       4988       4988          0    1512441          0      34088
 10   15224642       3818          0          0         25         25       1884       1884          0          0          0      21218
 11    4023142     603616          0          0          2          2        555        555          0     602502          0      30177
 12    5915692       1460          0          0          7          7        723        723          0          0          0      19261
 13    9709569       2380          0          0         14         14       1176       1176          0          0          0      33030
 14    4832393       1204          0          0          3          3        599        599          0          0          0      22476
 15   31664039       7994          0          0         55         55       3942       3942          0          0          0      15558
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.457s (  956.451 blocks/s,  59.778 Mb/s)
Attachment 45: Screenshot_from_2024-06-10_15-02-52.png
Screenshot_from_2024-06-10_15-02-52.png
Attachment 46: Screenshot_from_2024-06-10_15-03-47.png
Screenshot_from_2024-06-10_15-03-47.png
Attachment 47: Screenshot_from_2024-06-10_15-04-15.png
Screenshot_from_2024-06-10_15-04-15.png
Attachment 48: Screenshot_from_2024-06-10_15-04-48.png
Screenshot_from_2024-06-10_15-04-48.png
Attachment 49: Screenshot_from_2024-06-10_15-05-18.png
Screenshot_from_2024-06-10_15-05-18.png
Attachment 50: Screenshot_from_2024-06-10_15-15-55.png
Screenshot_from_2024-06-10_15-15-55.png
  646   Sun Jun 9 22:49:56 2024 TD23:00-07:00 Monday 9 June
23.47 DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 3

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 4-5 & 9

      WR timestamp - attachment 7

      all system wide checks OK *except* attachments 6 & 10

      per FEE64 Rate spectra - attachment 8

      per FEE64 1.8.W spectra - 20us FSR - attachments 11-13

      Merger, TapeServer etc - attachments 14-15
       data file S181/R4, TapeServer No Storage mode

00.00 Beam ON 3s spill on, 2s spill off
      99%+ fission fragments
      ADC data item stats - attachment 16


02.03 Data storage enabled
 
      per FEE64 1.8.H spectra - attachments 17-18

02.35 analysis data file S181/R4_28 - attachment 19
      data file S181/R4
      max deadtime aida08 18%
      n+n FEE64s 12-18%, all p+n FEE64s < 2%

02.41 Noted that there is disc data written to disk - all disc data now disabled for all FEE64s

04.08 DSSSD bias & leakage current OK - attachment 20

      FEE64 temperatures OK - attachment 21

      ADC data item stats - attachment 22

      per FEE64 Rate spectra - attachment 23

04.15 aida02 no data
      restarted merger per https://elog.ph.ed.ac.uk/DESPEC/644

06.00 DSSSD bias & leakage current OK - attachment 24

      FEE64 temperatures OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Rate spectra - attachment 27
Attachment 1: Screenshot_from_2024-06-09_23-47-24.png
Screenshot_from_2024-06-09_23-47-24.png
Attachment 2: Screenshot_from_2024-06-09_23-47-42.png
Screenshot_from_2024-06-09_23-47-42.png
Attachment 3: Screenshot_from_2024-06-09_23-48-00.png
Screenshot_from_2024-06-09_23-48-00.png
Attachment 4: Screenshot_from_2024-06-09_23-48-04.png
Screenshot_from_2024-06-09_23-48-04.png
Attachment 5: Screenshot_from_2024-06-09_23-48-16.png
Screenshot_from_2024-06-09_23-48-16.png
Attachment 6: Screenshot_from_2024-06-09_23-48-41.png
Screenshot_from_2024-06-09_23-48-41.png
Attachment 7: Screenshot_from_2024-06-09_23-49-10.png
Screenshot_from_2024-06-09_23-49-10.png
Attachment 8: Screenshot_from_2024-06-09_23-49-18.png
Screenshot_from_2024-06-09_23-49-18.png
Attachment 9: Screenshot_from_2024-06-09_23-55-16.png
Screenshot_from_2024-06-09_23-55-16.png
Attachment 10: Screenshot_from_2024-06-09_23-55-48.png
Screenshot_from_2024-06-09_23-55-48.png
Attachment 11: Screenshot_from_2024-06-09_23-56-49.png
Screenshot_from_2024-06-09_23-56-49.png
Attachment 12: Screenshot_from_2024-06-09_23-57-05.png
Screenshot_from_2024-06-09_23-57-05.png
Attachment 13: Screenshot_from_2024-06-09_23-57-28.png
Screenshot_from_2024-06-09_23-57-28.png
Attachment 14: Screenshot_from_2024-06-09_23-58-23.png
Screenshot_from_2024-06-09_23-58-23.png
Attachment 15: Screenshot_from_2024-06-09_23-58-29.png
Screenshot_from_2024-06-09_23-58-29.png
Attachment 16: Screenshot_from_2024-06-10_00-11-59.png
Screenshot_from_2024-06-10_00-11-59.png
Attachment 17: Screenshot_from_2024-06-10_01-27-52.png
Screenshot_from_2024-06-10_01-27-52.png
Attachment 18: Screenshot_from_2024-06-10_02-01-38.png
Screenshot_from_2024-06-10_02-01-38.png
Attachment 19: R4_28
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  253194077 ( 1696553.0 Hz)
        Other data format:    8725923 (   58468.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1564 (      10.5 Hz)
                               RESUME:       1562 (      10.5 Hz)
                              SYNC100:      32556 (     218.1 Hz)
                              WR48-63:      32556 (     218.1 Hz)
                           FEE64 disc:    3047502 (   20420.1 Hz)
                             MBS info:    5610183 (   37591.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     916603 (    6141.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      149.240 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.269                0.000
  1               18.572                0.000
  2                0.203                0.000
  3               23.804                0.000
  4                0.695                0.000
  5               26.433                0.000
  6                0.080                0.000
  7               27.735                0.000
  8                0.165                0.000
  9                3.128                0.000
 10                0.210                0.000
 11                0.060                0.000
 12                0.056                0.000
 13                0.214                0.000
 14                0.073                0.000
 15                1.249                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8063714     229134          0          0         29         29       1100       1100     222403       4473          0      40546
  1   19451828     341670          0          0        189        189       2480       2480     336332          0          0      50938
  2    9906047    1695573          0          0         32         32       1474       1474     189984    1502577          0      57685
  3   25485973     434336          0          0        375        375       3209       3209     427168          0          0      33926
  4    5225989    1088546          0          0         26         25        750        750      83786    1003209          0      36956
  5   23827850     287577          0          0        303        303       2970       2970     281031          0          0      42461
  6    7632945     803643          0          0         10         10       1067       1067     163557     637932          0      57385
  7   43365063     397279          0          0        289        288       5505       5505     385692          0          0     246600
  8    9556580     143268          0          0         16         16       1175       1175     140886          0          0      58513
  9   31955982    1616616          0          0        155        155       4212       4212     145847    1462035          0      55475
 10   13350940      89761          0          0         23         23       1645       1645      86425          0          0      31430
 11    5509660    1131598          0          0          8          8        744        744     130137     999957          0      52425
 12    7108103     102578          0          0          8          8        887        887     100788          0          0      34624
 13   10337974     150173          0          0         26         26       1313       1313     147495          0          0      56686
 14    5578272     207359          0          0          7          7        687        687     205971          0          0      33046
 15   26837157       6812          0          0         68         68       3338       3338          0          0          0      27907
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   38.183s (  838.078 blocks/s,  52.380 Mb/s)
Attachment 20: Screenshot_from_2024-06-10_04-08-10.png
Screenshot_from_2024-06-10_04-08-10.png
Attachment 21: Screenshot_from_2024-06-10_04-08-44.png
Screenshot_from_2024-06-10_04-08-44.png
Attachment 22: Screenshot_from_2024-06-10_04-10-54.png
Screenshot_from_2024-06-10_04-10-54.png
Attachment 23: Screenshot_from_2024-06-10_04-11-33.png
Screenshot_from_2024-06-10_04-11-33.png
Attachment 24: Screenshot_from_2024-06-10_06-00-15.png
Screenshot_from_2024-06-10_06-00-15.png
Attachment 25: Screenshot_from_2024-06-10_06-00-52.png
Screenshot_from_2024-06-10_06-00-52.png
Attachment 26: Screenshot_from_2024-06-10_06-01-18.png
Screenshot_from_2024-06-10_06-01-18.png
Attachment 27: Screenshot_from_2024-06-10_06-01-43.png
Screenshot_from_2024-06-10_06-01-43.png
  645   Sun Jun 9 16:38:30 2024 Norah , Muneerah, JB16:0-00:00 9 June 2024

AIDA02 and AIDA06 gave zero attachment 1. After connecting with Tom to fix it, now it works.

17:00

DSSSD bias & leakage current - attachment 2

FEE64 temperatures OK - attachment 3

Statistics  attachment 4

 

17:39

Most of AIDA0 gave zero. I followed the instructions that Tom gave me to fix it, and now they work.

23:19 Flange removed. Starting to take beam.

 

 

 

 

Attachment 1: Screenshot_from_2024-06-09_17-24-13.png
Screenshot_from_2024-06-09_17-24-13.png
Attachment 2: Screenshot_from_2024-06-09_18-00-27.png
Screenshot_from_2024-06-09_18-00-27.png
Attachment 3: Screenshot_from_2024-06-09_18-01-49.png
Screenshot_from_2024-06-09_18-01-49.png
Attachment 4: Screenshot_from_2024-06-09_18-01-23.png
Screenshot_from_2024-06-09_18-01-23.png
  644   Sun Jun 9 13:32:48 2024 TD[How To] Recover DAQ when one FEE64 stops sending data - short(er) version
1) Select 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode please ignore this step

2) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

3) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

4) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

5) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

6) Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

8) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

9) Unselect 'No Storage' [workspace 3, display 1, browser tab 'TapeServer']
    If already in 'No Storage' mode at the start of this procedure please ignore this step


   If you continue to have problems, please request expert assistance.
  643   Sat Jun 8 23:00:52 2024 TDSunday 9 June
23.58 8.6.24

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

03.43 slow comparator 0x64

      DSSSD bias & leakage current OK - attachments 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7

      per FEE64 1.8.H spectra - attachments 8-9

      per FEE64 Rate spectra - attachment 10


09.11 DSSSD bias & leakage current OK - attachments 11

      FEE64 temperatures OK - attachment 12

      ADC data item stats - attachment 13

      per FEE64 Rate spectra - attachment 14


15.05 online spectra
      per DSSSD HEC E_p versus E_n and x versus y
      observe flat field of c. 500MeV at per spull rate c. 5Hz - fission fragments?
Attachment 1: Screenshot_from_2024-06-08_23-58-55.png
Screenshot_from_2024-06-08_23-58-55.png
Attachment 2: Screenshot_from_2024-06-08_23-59-16.png
Screenshot_from_2024-06-08_23-59-16.png
Attachment 3: Screenshot_from_2024-06-09_00-00-02.png
Screenshot_from_2024-06-09_00-00-02.png
Attachment 4: Screenshot_from_2024-06-09_00-00-32.png
Screenshot_from_2024-06-09_00-00-32.png
Attachment 5: Screenshot_from_2024-06-09_03-43-32.png
Screenshot_from_2024-06-09_03-43-32.png
Attachment 6: Screenshot_from_2024-06-09_03-44-19.png
Screenshot_from_2024-06-09_03-44-19.png
Attachment 7: Screenshot_from_2024-06-09_03-46-03.png
Screenshot_from_2024-06-09_03-46-03.png
Attachment 8: Screenshot_from_2024-06-09_03-46-37.png
Screenshot_from_2024-06-09_03-46-37.png
Attachment 9: Screenshot_from_2024-06-09_03-47-40.png
Screenshot_from_2024-06-09_03-47-40.png
Attachment 10: Screenshot_from_2024-06-09_03-49-06.png
Screenshot_from_2024-06-09_03-49-06.png
Attachment 11: Screenshot_from_2024-06-09_11-15-26.png
Screenshot_from_2024-06-09_11-15-26.png
Attachment 12: Screenshot_from_2024-06-09_11-27-22.png
Screenshot_from_2024-06-09_11-27-22.png
Attachment 13: Screenshot_from_2024-06-09_11-27-52.png
Screenshot_from_2024-06-09_11-27-52.png
Attachment 14: Screenshot_from_2024-06-09_11-28-19.png
Screenshot_from_2024-06-09_11-28-19.png
Attachment 15: Capture_2.PNG
Capture_2.PNG
Attachment 16: Capture.PNG
Capture.PNG
  642   Sat Jun 8 13:22:36 2024 TDSaturday 8 June
14.09 aida02 WR stall
      DAQ STOP
      restart Merger, Mrger SETUP, Merger GO
      DAQ GO

      all histograms & stats zero'd

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK - attachment 4

      WR timestamp before/after Merger restart - attachments 3 & 5

      all system wide checks OK *except* attachments 6-7

      ADC, PAUSE and MBS correlation scaler data item stats - attachments 8-10
       ADC data items rates high - source installed?

      per FEE64 Rate spectra - attachments 11-13

      per FEE64 1.8.W spectra - 20us FSR - attachments 14-15

      per p+n FEE64 1.8.L spectra - attachment 16
       aida09 pulser peak width 60 ch FWHM

      Merger, TapeServer etc - attachments 17-19
       data file S181/R2_249 switch TapeServer to No Storage mode

      per p+n FEE64 1.8.W spectra - 20us FSR - attachment 20

20.33 DSSSD bias & leakage current OK - attachments 21

      FEE64 temperatures OK - attachment 22

      ADC data item stats - attachment 23

      per FEE64 Rate spectra - attachment 24
Attachment 1: Screenshot_from_2024-06-08_14-07-37.png
Screenshot_from_2024-06-08_14-07-37.png
Attachment 2: Screenshot_from_2024-06-08_14-07-52.png
Screenshot_from_2024-06-08_14-07-52.png
Attachment 3: Screenshot_from_2024-06-08_14-08-47.png
Screenshot_from_2024-06-08_14-08-47.png
Attachment 4: Screenshot_from_2024-06-08_14-08-21.png
Screenshot_from_2024-06-08_14-08-21.png
Attachment 5: Screenshot_from_2024-06-08_14-11-30.png
Screenshot_from_2024-06-08_14-11-30.png
Attachment 6: Screenshot_from_2024-06-08_14-12-20.png
Screenshot_from_2024-06-08_14-12-20.png
Attachment 7: Screenshot_from_2024-06-08_14-12-25.png
Screenshot_from_2024-06-08_14-12-25.png
Attachment 8: Screenshot_from_2024-06-08_14-12-53.png
Screenshot_from_2024-06-08_14-12-53.png
Attachment 9: Screenshot_from_2024-06-08_14-13-12.png
Screenshot_from_2024-06-08_14-13-12.png
Attachment 10: Screenshot_from_2024-06-08_14-13-34.png
Screenshot_from_2024-06-08_14-13-34.png
Attachment 11: Screenshot_from_2024-06-08_14-14-41.png
Screenshot_from_2024-06-08_14-14-41.png
Attachment 12: Screenshot_from_2024-06-08_14-14-48.png
Screenshot_from_2024-06-08_14-14-48.png
Attachment 13: Screenshot_from_2024-06-08_14-14-54.png
Screenshot_from_2024-06-08_14-14-54.png
Attachment 14: Screenshot_from_2024-06-08_14-16-54.png
Screenshot_from_2024-06-08_14-16-54.png
Attachment 15: Screenshot_from_2024-06-08_14-17-49.png
Screenshot_from_2024-06-08_14-17-49.png
Attachment 16: Screenshot_from_2024-06-08_14-20-04.png
Screenshot_from_2024-06-08_14-20-04.png
Attachment 17: Screenshot_from_2024-06-08_14-21-31.png
Screenshot_from_2024-06-08_14-21-31.png
Attachment 18: Screenshot_from_2024-06-08_14-22-05.png
Screenshot_from_2024-06-08_14-22-05.png
Attachment 19: Screenshot_from_2024-06-08_14-22-12.png
Screenshot_from_2024-06-08_14-22-12.png
Attachment 20: Screenshot_from_2024-06-08_14-46-36.png
Screenshot_from_2024-06-08_14-46-36.png
Attachment 21: Screenshot_from_2024-06-08_20-33-56.png
Screenshot_from_2024-06-08_20-33-56.png
Attachment 22: Screenshot_from_2024-06-08_20-36-17.png
Screenshot_from_2024-06-08_20-36-17.png
Attachment 23: Screenshot_from_2024-06-08_20-36-50.png
Screenshot_from_2024-06-08_20-36-50.png
Attachment 24: Screenshot_from_2024-06-08_20-37-34.png
Screenshot_from_2024-06-08_20-37-34.png
  641   Fri Jun 7 16:08:37 2024 TDFriday 7 June
05.01 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      9x < 20k, max 228k aida08

      per FEE64 Rate spectra - attachment 4


08.18 DSSSD bias & leakage current - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats - attachment 7
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 8


14.37 DSSSD bias & leakage current - attachment 9

      FEE64 temperatures OK - attachment 10

      ADC data item stats - attachment 11
      9x < 20k, max 224k aida08

      per FEE64 Rate spectra - attachment 12

      Writing to disk file S181/R1 (continuing alpha background file)

14.38 End run S181/R1 and start new run file S181/R2

17.09 08.18 DSSSD bias & leakage current - attachment 13

      FEE64 temperatures OK - attachment 14

      MBS correlation scaler data - attachment 15
       data to aida01, aida02, aida03 and aida10

      ADC data item stats - attachment 16
      9x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 17

      Merger/TapeServer etc - attachments 18-19 
Attachment 1: Screenshot_from_2024-06-07_05-01-53.png
Screenshot_from_2024-06-07_05-01-53.png
Attachment 2: Screenshot_from_2024-06-07_05-00-13.png
Screenshot_from_2024-06-07_05-00-13.png
Attachment 3: Screenshot_from_2024-06-07_05-00-49.png
Screenshot_from_2024-06-07_05-00-49.png
Attachment 4: Screenshot_from_2024-06-07_05-01-23.png
Screenshot_from_2024-06-07_05-01-23.png
Attachment 5: Screenshot_from_2024-06-07_08-18-34.png
Screenshot_from_2024-06-07_08-18-34.png
Attachment 6: Screenshot_from_2024-06-07_08-19-54.png
Screenshot_from_2024-06-07_08-19-54.png
Attachment 7: Screenshot_from_2024-06-07_08-20-21.png
Screenshot_from_2024-06-07_08-20-21.png
Attachment 8: Screenshot_from_2024-06-07_08-19-22.png
Screenshot_from_2024-06-07_08-19-22.png
Attachment 9: Screenshot_from_2024-06-07_14-38-06.png
Screenshot_from_2024-06-07_14-38-06.png
Attachment 10: Screenshot_from_2024-06-07_14-36-36.png
Screenshot_from_2024-06-07_14-36-36.png
Attachment 11: Screenshot_from_2024-06-07_14-37-12.png
Screenshot_from_2024-06-07_14-37-12.png
Attachment 12: Screenshot_from_2024-06-07_14-37-42.png
Screenshot_from_2024-06-07_14-37-42.png
Attachment 13: Screenshot_from_2024-06-07_17-09-16.png
Screenshot_from_2024-06-07_17-09-16.png
Attachment 14: Screenshot_from_2024-06-07_17-09-46.png
Screenshot_from_2024-06-07_17-09-46.png
Attachment 15: Screenshot_from_2024-06-07_17-10-10.png
Screenshot_from_2024-06-07_17-10-10.png
Attachment 16: Screenshot_from_2024-06-07_17-10-52.png
Screenshot_from_2024-06-07_17-10-52.png
Attachment 17: Screenshot_from_2024-06-07_17-11-15.png
Screenshot_from_2024-06-07_17-11-15.png
Attachment 18: Screenshot_from_2024-06-07_17-11-40.png
Screenshot_from_2024-06-07_17-11-40.png
Attachment 19: Screenshot_from_2024-06-07_17-12-21.png
Screenshot_from_2024-06-07_17-12-21.png
  640   Thu Jun 6 11:13:29 2024 TDThursday 6 June
12.09 DSSSD bias & leakage current - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
      8x < 20k, max 247k aida08

      per FEE64 Rate spectra - attachment 4

12.16 all histograms & stats zero'd


21.08 DSSSD bias & leakage current - attachments 5-6

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 8
      8x < 20k, max 230k aida08

      per FEE64 Rate spectra - attachment 9


      
Attachment 1: Screenshot_from_2024-06-06_12-09-27.png
Screenshot_from_2024-06-06_12-09-27.png
Attachment 2: Screenshot_from_2024-06-06_12-11-58.png
Screenshot_from_2024-06-06_12-11-58.png
Attachment 3: Screenshot_from_2024-06-06_12-12-38.png
Screenshot_from_2024-06-06_12-12-38.png
Attachment 4: Screenshot_from_2024-06-06_12-13-09.png
Screenshot_from_2024-06-06_12-13-09.png
Attachment 5: Screenshot_from_2024-06-06_21-08-07.png
Screenshot_from_2024-06-06_21-08-07.png
Attachment 6: Screenshot_from_2024-06-06_21-08-38.png
Screenshot_from_2024-06-06_21-08-38.png
Attachment 7: Screenshot_from_2024-06-06_21-09-15.png
Screenshot_from_2024-06-06_21-09-15.png
Attachment 8: Screenshot_from_2024-06-06_21-10-02.png
Screenshot_from_2024-06-06_21-10-02.png
Attachment 9: Screenshot_from_2024-06-06_21-10-26.png
Screenshot_from_2024-06-06_21-10-26.png
  639   Wed Jun 5 08:34:47 2024 JB. AM, MP, MYDetector bias tests

We are trying to bias each system on its own and in sequence biasing each other detector system to see the interdependence.
 

HV OK. TEMP OK. Attachment 1 &2.

9.43 bPlast unbiased, BB7 unbiased. Attachments 3-6 is the noise situation with only AIDA biased (bPlast was biased but turned off after AIDA was biased.)

10/16 FEEs < 20k noise. Max noise is 250k. Downstream AIDA appears to be seeing additional noise since S100.


11:21 Returning after an hour the noise has increased significantly in many channels. Scale changed to 10000 in histograms, it appears that the problem is from a couple hot channels.

11:41 'Power' cycled AIDA turned off. bPlast turned on, on its own, then BB7. bPlast doesn't seem to notice anything.

12:21 bPlast and BB7 on. AIDA ON. Rates and histograms given by attachment 7-8. More FEEs noisy, but same level as 11:21. There appears to be the same noisy channel in all of the FEEs (?) Some grounding not properly accounted for, or a wire touching somewhere (?)

13:55h: AIDA & BB7 on. Rates and histograms given in attachments 9-10. Same level as bplast & AIDA on. Same noisy channels in FEEs. According to Nic, these correspond to the end of the ribbon cables & is normal.

13:59 AIDA, bPlast & BB7 on before lunch. Same levels of noise as in previous combinations (attachments 11-12).

So it seems we can run with the three sub-systems biased simultaneously.

 

 

Attachment 1: Screenshot_from_2024-06-05_09-36-25.png
Screenshot_from_2024-06-05_09-36-25.png
Attachment 2: Screenshot_from_2024-06-05_09-39-42.png
Screenshot_from_2024-06-05_09-39-42.png
Attachment 3: Screenshot_from_2024-06-05_09-41-44.png
Screenshot_from_2024-06-05_09-41-44.png
Attachment 4: Screenshot_from_2024-06-05_09-41-52.png
Screenshot_from_2024-06-05_09-41-52.png
Attachment 5: Screenshot_from_2024-06-05_09-42-35.png
Screenshot_from_2024-06-05_09-42-35.png
Attachment 6: Screenshot_from_2024-06-05_09-43-43.png
Screenshot_from_2024-06-05_09-43-43.png
Attachment 7: Screenshot_from_2024-06-05_11-40-33.png
Screenshot_from_2024-06-05_11-40-33.png
Attachment 8: Screenshot_from_2024-06-05_11-40-39.png
Screenshot_from_2024-06-05_11-40-39.png
Attachment 9: Screenshot_from_2024-06-05_12-19-14.png
Screenshot_from_2024-06-05_12-19-14.png
Attachment 10: Screenshot_from_2024-06-05_12-20-42.png
Screenshot_from_2024-06-05_12-20-42.png
Attachment 11: Screenshot_from_2024-06-05_13-58-21.png
Screenshot_from_2024-06-05_13-58-21.png
Attachment 12: Screenshot_from_2024-06-05_13-55-08.png
Screenshot_from_2024-06-05_13-55-08.png
Attachment 13: Screenshot_from_2024-06-05_12-20-42.png
Screenshot_from_2024-06-05_12-20-42.png
Attachment 14: Screenshot_from_2024-06-05_12-19-14.png
Screenshot_from_2024-06-05_12-19-14.png
  638   Tue Jun 4 15:40:32 2024 NHTues 4 Jun
DEGAS Array over Snout again

Situation similar to yesterday, most p+n rates good
Attachment 1: Screenshot_from_2024-06-04_16-39-28.png
Screenshot_from_2024-06-04_16-39-28.png
Attachment 2: Screenshot_from_2024-06-04_16-39-47.png
Screenshot_from_2024-06-04_16-39-47.png
  637   Mon Jun 3 14:40:34 2024 JB, NH03 June 2024

15:40 reseated FEE64s adaptors of aida01, aida09, aida13 & aida16.

TEMP OK. HV OK. System wide checks all passed excl. aida07 ADC calibration, see attached.

Histograms and rates look same as previous, reseating aida01 seems to have worked. aida16 still unhappy.

re-reseated aida16. 16:19 - appears to have fixed the problem in the channel.

Pulser test + connected.

bPlast ground removed from frame.

To-do:

- Need clarification about links connected on AIDA n+n FEE64 adaptor boards. Currently, the situation is given by the attachment 8 & 9.

- Fix BB7 grounding, continuity between snout and frame observed.

- Pulser walkthrough for n+n, can do p+n when convienient.

 

 

18.30 Noted FEE64 ADC stats now significantly higher ( attachments 9-10 ) cf. earlier this afternoon (attachments 1-2)

DSSSD bias OFF. FEE64 power OFF.

 

 

Attachment 1: Screenshot_from_2024-06-03_15-49-35.png
Screenshot_from_2024-06-03_15-49-35.png
Attachment 2: Screenshot_from_2024-06-03_15-49-24.png
Screenshot_from_2024-06-03_15-49-24.png
Attachment 3: Screenshot_from_2024-06-03_15-45-54.png
Screenshot_from_2024-06-03_15-45-54.png
Attachment 4: Screenshot_from_2024-06-03_15-42-31.png
Screenshot_from_2024-06-03_15-42-31.png
Attachment 5: Screenshot_from_2024-06-03_15-42-00.png
Screenshot_from_2024-06-03_15-42-00.png
Attachment 6: Screenshot_from_2024-06-03_16-18-49.png
Screenshot_from_2024-06-03_16-18-49.png
Attachment 7: 20240603_172243.jpg
20240603_172243.jpg
Attachment 8: 20240603_172233.jpg
20240603_172233.jpg
Attachment 9: Screenshot_from_2024-06-03_18-34-14.png
Screenshot_from_2024-06-03_18-34-14.png
Attachment 10: Screenshot_from_2024-06-03_18-34-30.png
Screenshot_from_2024-06-03_18-34-30.png
  636   Mon Jun 3 07:35:56 2024 TDMonday 3 June
08.28 DSSSD bias & leakage current - attachments 1-2

      FEE64 temps OK - attachment 3

      ADC data item stats - attachment 4

      per FEE64 Stat spectra - attachment 5

      per FEE64 1.8.L spectra - attachment 6

      per FEE64 1.8.W spectra - 0us FSR - attachments 7-8

08.39 DAQ stops file S181/R1_36

      FEE64 power OFF
      DSSSD bias ON

11.12 DSSSD bias OFF
Attachment 1: Screenshot_from_2024-06-03_08-27-39.png
Screenshot_from_2024-06-03_08-27-39.png
Attachment 2: Screenshot_from_2024-06-03_08-28-04.png
Screenshot_from_2024-06-03_08-28-04.png
Attachment 3: Screenshot_from_2024-06-03_08-28-37.png
Screenshot_from_2024-06-03_08-28-37.png
Attachment 4: Screenshot_from_2024-06-03_08-29-04.png
Screenshot_from_2024-06-03_08-29-04.png
Attachment 5: Screenshot_from_2024-06-03_08-29-47.png
Screenshot_from_2024-06-03_08-29-47.png
Attachment 6: Screenshot_from_2024-06-03_08-30-52.png
Screenshot_from_2024-06-03_08-30-52.png
Attachment 7: Screenshot_from_2024-06-03_08-34-18.png
Screenshot_from_2024-06-03_08-34-18.png
Attachment 8: Screenshot_from_2024-06-03_08-35-27.png
Screenshot_from_2024-06-03_08-35-27.png
  635   Sun Jun 2 01:45:05 2024 TDSunday 2 June
02.41 DSSSD bias & leakage current - attachment 1

      FEE64 temps OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Stat spectra - attachment 4


06.52 DSSSD bias & leakage current - attachment 5

      FEE64 temps OK - attachment 6

      ADC data item stats - attachment 7

      per FEE64 Stat spectra - attachment 8

12.13 DSSSD bias & leakage current - attachment 9

      FEE64 temps OK - attachment 10

      ADC data item stats - attachment 11

      per FEE64 Stat spectra - attachment 12


14.51 DSSSD bias & leakage current - attachments 13-14

      FEE64 temps OK - attachment 15

      ADC data item stats - attachment 16

      per FEE64 Stat spectra - attachment 17

21.49 DSSSD bias & leakage current - attachment 18

      FEE64 temps OK - attachment 19

      ADC data item stats - attachment 20

      per FEE64 Stat spectra - attachment 21


03.55 DSSSD bias & leakage current - attachment 22

      FEE64 temps OK - attachment 23

      ADC data item stats - attachment 24

      per FEE64 Stat spectra - attachment 25
Attachment 1: Screenshot_from_2024-06-02_02-40-53.png
Screenshot_from_2024-06-02_02-40-53.png
Attachment 2: Screenshot_from_2024-06-02_02-41-25.png
Screenshot_from_2024-06-02_02-41-25.png
Attachment 3: Screenshot_from_2024-06-02_02-41-49.png
Screenshot_from_2024-06-02_02-41-49.png
Attachment 4: Screenshot_from_2024-06-02_02-42-57.png
Screenshot_from_2024-06-02_02-42-57.png
Attachment 5: Screenshot_from_2024-06-02_06-52-56.png
Screenshot_from_2024-06-02_06-52-56.png
Attachment 6: Screenshot_from_2024-06-02_06-53-29.png
Screenshot_from_2024-06-02_06-53-29.png
Attachment 7: Screenshot_from_2024-06-02_06-53-57.png
Screenshot_from_2024-06-02_06-53-57.png
Attachment 8: Screenshot_from_2024-06-02_06-54-52.png
Screenshot_from_2024-06-02_06-54-52.png
Attachment 9: Screenshot_from_2024-06-02_12-13-13.png
Screenshot_from_2024-06-02_12-13-13.png
Attachment 10: Screenshot_from_2024-06-02_12-13-46.png
Screenshot_from_2024-06-02_12-13-46.png
Attachment 11: Screenshot_from_2024-06-02_12-14-07.png
Screenshot_from_2024-06-02_12-14-07.png
Attachment 12: Screenshot_from_2024-06-02_12-15-07.png
Screenshot_from_2024-06-02_12-15-07.png
Attachment 13: Screenshot_from_2024-06-02_14-47-23.png
Screenshot_from_2024-06-02_14-47-23.png
Attachment 14: Screenshot_from_2024-06-02_14-51-12.png
Screenshot_from_2024-06-02_14-51-12.png
Attachment 15: Screenshot_from_2024-06-02_14-51-45.png
Screenshot_from_2024-06-02_14-51-45.png
Attachment 16: Screenshot_from_2024-06-02_14-52-12.png
Screenshot_from_2024-06-02_14-52-12.png
Attachment 17: Screenshot_from_2024-06-02_14-52-46.png
Screenshot_from_2024-06-02_14-52-46.png
Attachment 18: Screenshot_from_2024-06-02_21-47-35.png
Screenshot_from_2024-06-02_21-47-35.png
Attachment 19: Screenshot_from_2024-06-02_21-48-05.png
Screenshot_from_2024-06-02_21-48-05.png
Attachment 20: Screenshot_from_2024-06-02_21-48-30.png
Screenshot_from_2024-06-02_21-48-30.png
Attachment 21: Screenshot_from_2024-06-02_21-49-08.png
Screenshot_from_2024-06-02_21-49-08.png
Attachment 22: Screenshot_from_2024-06-03_03-55-12.png
Screenshot_from_2024-06-03_03-55-12.png
Attachment 23: Screenshot_from_2024-06-03_03-55-56.png
Screenshot_from_2024-06-03_03-55-56.png
Attachment 24: Screenshot_from_2024-06-03_03-56-23.png
Screenshot_from_2024-06-03_03-56-23.png
Attachment 25: Screenshot_from_2024-06-03_03-58-23.png
Screenshot_from_2024-06-03_03-58-23.png
  634   Sat Jun 1 12:48:54 2024 TDSaturday 1 June
13.26 DSSSD bias & leakage current - attachments 1-3

      FEE64 temps OK - attachment 4
       aida02 asic sensor u/s

      All system wide checks OK *except* WR decoder status - attachment 5

      ADC data item stats - attachment 6

      per FEE64 Rate spectra - attachments 7-8

      per FEE64 1.8.W spectra - 20us FSR - attachments 9-10

      WR timestamps OK - attachment 11

13.59 Data file S181/R1
      alpha background
      BNC PB-5 disconnected?
      slow comparator 0x64

      ADC data item stats - attachment 12
      Merger, TapeServer etc - attachments 13-14

14.02 All histograms & stats zero'd



16.19 DSSSD bias & leakage current - attachment 15

      FEE64 temps OK - attachment 16

      ADC data item stats - attachment 17

      per FEE64 Stat spectra - attachment 18


19.16 DSSSD bias & leakage current - attachments 19-20

      FEE64 temps OK - attachment 21

      ADC data item stats - attachment 22

      per FEE64 Stat spectra - attachment 23

      2x Check ASIC Control


21.42 DSSSD bias & leakage current - attachment 24

      FEE64 temps OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Stat spectra - attachment 27

00.48 DSSSD bias & leakage current - attachment 28

      FEE64 temps OK - attachment 29

      ADC data item stats - attachment 30

      per FEE64 Stat spectra - attachment 31
Attachment 1: Screenshot_from_2024-06-01_13-26-48.png
Screenshot_from_2024-06-01_13-26-48.png
Attachment 2: Screenshot_from_2024-06-01_13-27-24.png
Screenshot_from_2024-06-01_13-27-24.png
Attachment 3: Screenshot_from_2024-06-01_13-27-47.png
Screenshot_from_2024-06-01_13-27-47.png
Attachment 4: Screenshot_from_2024-06-01_13-40-22.png
Screenshot_from_2024-06-01_13-40-22.png
Attachment 5: Screenshot_from_2024-06-01_13-43-09.png
Screenshot_from_2024-06-01_13-43-09.png
Attachment 6: Screenshot_from_2024-06-01_13-44-41.png
Screenshot_from_2024-06-01_13-44-41.png
Attachment 7: Screenshot_from_2024-06-01_13-45-21.png
Screenshot_from_2024-06-01_13-45-21.png
Attachment 8: Screenshot_from_2024-06-01_13-45-49.png
Screenshot_from_2024-06-01_13-45-49.png
Attachment 9: Screenshot_from_2024-06-01_13-47-30.png
Screenshot_from_2024-06-01_13-47-30.png
Attachment 10: Screenshot_from_2024-06-01_13-48-20.png
Screenshot_from_2024-06-01_13-48-20.png
Attachment 11: Screenshot_from_2024-06-01_13-51-48.png
Screenshot_from_2024-06-01_13-51-48.png
Attachment 12: Screenshot_from_2024-06-01_13-53-14.png
Screenshot_from_2024-06-01_13-53-14.png
Attachment 13: Screenshot_from_2024-06-01_14-00-38.png
Screenshot_from_2024-06-01_14-00-38.png
Attachment 14: Screenshot_from_2024-06-01_14-01-08.png
Screenshot_from_2024-06-01_14-01-08.png
Attachment 15: Screenshot_from_2024-06-01_16-19-23.png
Screenshot_from_2024-06-01_16-19-23.png
Attachment 16: Screenshot_from_2024-06-01_16-27-28.png
Screenshot_from_2024-06-01_16-27-28.png
Attachment 17: Screenshot_from_2024-06-01_16-28-06.png
Screenshot_from_2024-06-01_16-28-06.png
Attachment 18: Screenshot_from_2024-06-01_16-29-07.png
Screenshot_from_2024-06-01_16-29-07.png
Attachment 19: Screenshot_from_2024-06-01_19-18-17.png
Screenshot_from_2024-06-01_19-18-17.png
Attachment 20: Screenshot_from_2024-06-01_19-15-57.png
Screenshot_from_2024-06-01_19-15-57.png
Attachment 21: Screenshot_from_2024-06-01_19-18-48.png
Screenshot_from_2024-06-01_19-18-48.png
Attachment 22: Screenshot_from_2024-06-01_19-19-11.png
Screenshot_from_2024-06-01_19-19-11.png
Attachment 23: Screenshot_from_2024-06-01_19-20-03.png
Screenshot_from_2024-06-01_19-20-03.png
Attachment 24: Screenshot_from_2024-06-01_21-41-48.png
Screenshot_from_2024-06-01_21-41-48.png
Attachment 25: Screenshot_from_2024-06-01_21-42-24.png
Screenshot_from_2024-06-01_21-42-24.png
Attachment 26: Screenshot_from_2024-06-01_21-42-49.png
Screenshot_from_2024-06-01_21-42-49.png
Attachment 27: Screenshot_from_2024-06-01_21-43-54.png
Screenshot_from_2024-06-01_21-43-54.png
Attachment 28: Screenshot_from_2024-06-02_00-48-44.png
Screenshot_from_2024-06-02_00-48-44.png
Attachment 29: Screenshot_from_2024-06-02_00-49-22.png
Screenshot_from_2024-06-02_00-49-22.png
Attachment 30: Screenshot_from_2024-06-02_00-49-48.png
Screenshot_from_2024-06-02_00-49-48.png
Attachment 31: Screenshot_from_2024-06-02_00-50-53.png
Screenshot_from_2024-06-02_00-50-53.png
  633   Mon May 27 17:20:49 2024 TDMonday 27 May
17.50 bPlas cabling, grounds etc connected, power OFF
       CC reports that bPlas power is actually ON

      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ASIC settings 2024Apr26-01.29.53
       p+n FFEE64 slow comparator 0xa, n+n Ohmic FEE64 0xf

      System wide checks OK *except* aida02 WR decoder status

      ADC data item stats - attachment 3
       9/16 < 20k, max 234k (aida08)

      per FEE64 Rate spectra - attachments 4-6

      per FEE64 1.8.W spectra - 20us FSR - attachments 7-8

      WR timestamps OK - attachment 9


18.50 FEE64 power OFF
Attachment 1: Screenshot_from_2024-05-27_18-09-02.png
Screenshot_from_2024-05-27_18-09-02.png
Attachment 2: Screenshot_from_2024-05-27_18-09-34.png
Screenshot_from_2024-05-27_18-09-34.png
Attachment 3: Screenshot_from_2024-05-27_18-10-20.png
Screenshot_from_2024-05-27_18-10-20.png
Attachment 4: Screenshot_from_2024-05-27_18-12-21.png
Screenshot_from_2024-05-27_18-12-21.png
Attachment 5: Screenshot_from_2024-05-27_18-13-15.png
Screenshot_from_2024-05-27_18-13-15.png
Attachment 6: Screenshot_from_2024-05-27_18-14-11.png
Screenshot_from_2024-05-27_18-14-11.png
Attachment 7: Screenshot_from_2024-05-27_18-18-40.png
Screenshot_from_2024-05-27_18-18-40.png
Attachment 8: Screenshot_from_2024-05-27_18-19-50.png
Screenshot_from_2024-05-27_18-19-50.png
Attachment 9: Screenshot_from_2024-05-27_18-27-26.png
Screenshot_from_2024-05-27_18-27-26.png
  632   Wed May 22 20:10:19 2024 NH, CC, HMA, MV, ZC, KZAIDA Snout Remounted
AIDA and the Snout have been remounted in S4

I had a check of the AIDA connectors and all looked good, the kapton PCB connectors were sitting comfortably above the interstage and I saw no stressed/loose connectors.
Multimeter showed no continuity between snout or inter-stage and AIDA ground

We put a single layer of wide black tape inside the end-cap to improve the light seal and minimise reflections

After mounting the snout I managed to bias both DSSDs to 100 V with no issues. I had to raise the DSSD#1 (Ch0) limit to 40 uA because it rises quite high during ramp up (then settles).

It is very hot in S4 (28 C) and the roof is closed for high-radiation experiments, so I expect it to get hotter. DSSD leakage currents were similar to before (20 and 15 uA)

There is also the new Temp and Humidity monitor, which you can view on AnyDesk at 192.168.11.119 (I think), there is a firefox window open already 
Attachment 1: 20240522_153750.jpg
20240522_153750.jpg
Attachment 2: 20240522_153752.jpg
20240522_153752.jpg
Attachment 3: 20240522_153755.jpg
20240522_153755.jpg
Attachment 4: 20240522_153757.jpg
20240522_153757.jpg
Attachment 5: 20240522_153805.jpg
20240522_153805.jpg
Attachment 6: 20240522_153807.jpg
20240522_153807.jpg
Attachment 7: 20240522_153809.jpg
20240522_153809.jpg
Attachment 8: 20240522_153810.jpg
20240522_153810.jpg
Attachment 9: 20240522_153819.jpg
20240522_153819.jpg
  631   Thu May 9 09:08:26 2024 TDCurrent NewMerger install - aida-gsi
[npg@aidas-gsi MIDAS]$ ls -l
total 524
drwxrwxr-x.  2 npg npg    156 Mar 17  2021 Aida
drwxrwxr-x.  2 npg npg   4096 Jul 13  2015 bin_Linux64
drwxrwxr-x.  4 npg npg     38 Jul  3  2018 config
drwxr-xr-x.  3 npg npg     22 Jan 23  2019 config_ORIG
drwxr-xr-x.  3 npg npg     17 Jan 23  2019 Data_Acq
drwxrwxr-x. 25 npg npg   4096 Aug 19  2020 DataPackage
drwxrwxr-x.  3 npg npg     25 Aug 29  2019 DB
drwxr-xr-x.  3 npg npg     25 Jan 23  2019 DB_ORIG
lrwxrwxrwx.  1 npg npg     14 May 30  2019 Embedded -> /home/Embedded
drwxrwxr-x.  2 npg npg   4096 Mar 17  2021 ISOL_Aida
drwxrwxr-x.  2 npg npg   4096 Nov 29  2018 ISOL_Aida_PreApr9
drwxrwxr-x.  7 npg npg     87 Jul  8  2009 lib
drwxrwxr-x.  2 npg npg   4096 Sep 19  2012 lib_Linux64
drwxr-xr-x.  5 npg npg     47 Jan 23  2019 Linux
drwxrwxr-x.  6 npg npg     58 Feb 22  2018 linux-ppc_4xx
drwxr-xr-x.  6 npg npg     58 Jan 23  2019 linux-ppc_4xx_ORIG
drwxrwxr-x.  3 npg npg     73 Feb 16  2022 log
-rwxrw-r--.  1 npg npg  12862 Jul  9  2009 midasicon.ico
-rwxrw-r--.  1 npg npg    773 Jul 23  2012 midasicon.png
-rwxrwxr-x.  1 npg npg    772 Jan 20  2019 MIDASNewGen_BuildAIDA.txt
-rwxrwxr--.  1 npg npg   2532 Jan 19  2019 MIDASNewGen_BuildCore.txt
drwxrwxr-x. 14 npg npg   4096 Feb 22  2018 MIDASpackages
drwxrwxr-x.  6 npg npg    124 May 14  2022 NewMerger
-rw-rw-r--.  1 npg npg 118873 Oct 18  2019 NewMergerdiff.log
-rw-rw-r--.  1 npg npg 118971 Oct 18  2019 NewMergerdiff.log~
drwxrwxr-x.  5 npg npg     52 Jan  8  2019 NewMerger_PreApr9
drwxrwxr-x.  6 npg npg    124 Mar 12  2020 NewMerger_PreMar21
drwxrwxr-x.  7 npg npg    104 Oct 17  2019 NewMerger_PreOct19
-rw-r--r--.  1 npg npg 230517 Mar 12  2020 NewMerger.tgz
drwxr-xr-x. 14 npg npg   4096 Jan 23  2019 OLD
drwxr-xr-x.  3 npg npg     17 Jan 23  2019 SOAP
drwxrwxrwx.  4 npg npg     67 Aug 19  2021 SpecData
drwxrwxr-x.  3 npg npg     21 Nov 15  2019 TapeServer
drwxr-xr-x.  3 npg npg     21 Jan 23  2019 TapeServer_PreFeb21
drwxrwxr-x.  3 npg npg     21 Nov 15  2019 TapeServer_PreMar21
drwxr-xr-x.  2 npg npg     44 Feb  2  2021 tcl
drwxr-xr-x. 12 npg npg    152 Jan 23  2019 TclHttpd
drwxrwxr-x. 11 npg npg    155 Oct 26  2017 TclTkpackages
lrwxrwxrwx.  1 npg npg     26 May 30  2019 XilinxLinux -> /home/Embedded/XilinxLinux

[npg@aidas-gsi NewMerger]$ ls -lR
.:
total 32
-rw-rw-r--. 1 npg npg 15096 Apr 16  2019 Aida_ASICreadout.c
drwxrwxr-x. 3 npg npg    23 Oct 18  2019 Html
drwxrwxr-x. 2 npg npg   172 May 14  2022 include
drwxrwxr-x. 8 npg npg   256 May 14  2022 MergeServer
-rw-rw-r--. 1 npg npg 11048 Aug 29  2019 R3B_readout.c
-rw-rw-r--. 1 npg npg   152 Oct 17  2019 README
drwxrwxr-x. 2 npg npg    34 Apr 29  2021 tcl

./Html:
total 0
drwxrwxr-x. 6 npg npg 187 Oct 13  2020 NewMerger

./Html/NewMerger:
total 60
lrwxrwxrwx. 1 npg npg    21 Oct 18  2019 Html -> /MIDAS/NewMerger/Html
drwxrwxr-x. 2 npg npg     6 Aug 26  2019 LinkRates
-rw-rw-r--. 1 npg npg  1210 Aug 26  2019 NewMerger.js
-rw-rw-r--. 1 npg npg  9417 Aug 29  2019 NewMerger.tcl
-rw-rw-r--. 1 npg npg  9332 Aug 26  2019 NewMerger.tcl~
-rw-rw-r--. 1 npg npg 13052 Aug 27  2019 NewMerger.tml
-rw-rw-r--. 1 npg npg 13851 Aug 26  2019 NewMerger.tml~
drwxrwxr-x. 2 npg npg    82 Oct 13  2020 Options
drwxrwxr-x. 2 npg npg   102 Oct 13  2020 RunControl
drwxrwxr-x. 2 npg npg   192 Aug 29  2019 Stats

./Html/NewMerger/LinkRates:
total 0

./Html/NewMerger/Options:
total 24
-rwxrwxr-x. 1 npg npg  725 Aug 26  2019 Options.js
-rwxrwxr-x. 1 npg npg 3934 Aug 26  2019 Options.tcl
-rwxrwxr-x. 1 npg npg 7864 Aug 27  2019 Options.tml
-rw-rw-r--. 1 npg npg 7864 Aug 27  2019 Options.tml~

./Html/NewMerger/RunControl:
total 44
-rw-rw-r--. 1 npg npg  4791 Aug 26  2019 implementation.tcl
-rw-rw-r--. 1 npg npg  1643 Aug 26  2019 stats.defn.tcl
-rw-rw-r--. 1 npg npg   539 Aug 26  2019 stats.tcl
-rw-rw-r--. 1 npg npg 12328 Aug 29  2019 sys.tcl
-rw-rw-r--. 1 npg npg 12231 Aug 26  2019 sys.tcl~

./Html/NewMerger/Stats:
total 52
-rw-rw-r--. 1 npg npg 2880 Aug 26  2019 MergeStatsALL.tcl
-rw-rw-r--. 1 npg npg 1013 Aug 26  2019 MergeStats.js
-rw-rw-r--. 1 npg npg 1534 Aug 26  2019 MergeStatsSINGLE.tcl
-rw-rw-r--. 1 npg npg 4259 Aug 26  2019 MergeStats.tcl
-rw-rw-r--. 1 npg npg 8467 Aug 27  2019 MergeStats.tml
-rw-rw-r--. 1 npg npg 8338 Aug 26  2019 MergeStats.tml~
-rw-rw-r--. 1 npg npg 1829 Aug 29  2019 stats.defn.tcl
-rw-rw-r--. 1 npg npg 1943 Aug 29  2019 stats.defn.tcl~

./include:
total 52
-rw-rw-r--. 1 npg npg 1079 Apr 16  2019 buffer.h
-rw-rw-r--. 1 npg npg 4473 Apr 16  2019 common.h
-rw-rw-r--. 1 npg npg 1199 Jul 16  2019 merge.h
-rw-rw-r--. 1 npg npg 1273 Apr 16  2019 message.h
-rw-rw-r--. 1 npg npg 1273 Oct 17  2019 message.h~
-rw-rw-r--. 1 npg npg 5288 Aug 29  2019 netvar.h
-rw-rw-r--. 1 npg npg 5256 Apr 16  2019 netvar.h~
-rw-rw-r--. 1 npg npg  912 Apr 16  2019 services.h
-rw-rw-r--. 1 npg npg  886 Apr 18  2019 stats.h
-rw-rw-r--. 1 npg npg 2710 Apr 16  2019 transfer.h

./MergeServer:
total 60
-rw-rw-r--. 1 npg npg  2606 Apr 16  2019 alloc_data_area.c
drwxr-xr-x. 2 npg npg    69 Aug 27  2019 bin64
drwxrwxr-x. 2 npg npg   128 Apr  6  2021 bin64.BackUp
-rw-rw-r--. 1 npg npg 10510 Sep 19  2019 buffer.c
-rwxrwxr-x. 1 npg npg   305 Apr 16  2019 build
drwxrwxr-x. 3 npg npg   279 May 14  2022 link
-rw-rw-r--. 1 npg npg   494 Apr 16  2019 Makefile
drwxrwxr-x. 3 npg npg   160 Apr  6  2021 master
drwxrwxr-x. 3 npg npg   143 Apr  6  2021 merge
-rw-rw-r--. 1 npg npg  3225 Sep 19  2019 message.c
-rw-rw-r--. 1 npg npg  3199 Sep 19  2019 netvar.c
-rwxrwxr-x. 1 npg npg   397 Aug 26  2019 run
-rw-rw-r--. 1 npg npg 15309 Sep 19  2019 signal.c
drwxrwxr-x. 3 npg npg   101 Sep 19  2019 statrate
-rw-rw-r--. 1 npg npg  1548 Sep 19  2019 stats.c
-rw-rw-r--. 1 npg npg   593 Apr 16  2019 version.c

./MergeServer/bin64:
total 148
-rwxr-xr-x. 1 npg npg 51964 May 11  2021 link64
-rwxr-xr-x. 1 npg npg 33312 May 11  2021 master64
-rwxr-xr-x. 1 npg npg 36027 May 11  2021 merge64
-rwxr-xr-x. 1 npg npg 21370 May 11  2021 statrate64

./MergeServer/bin64.BackUp:
total 256
-rwxr-xr-x. 1 npg npg 51948 Mar 16  2021 link64
-rwxrwxr-x. 1 npg npg 33184 Apr  6  2021 master64
-rwxr-xr-x. 1 npg npg 33312 Mar 16  2021 master64.new
-rwxr-xr-x. 1 npg npg 36027 Mar 16  2021 merge64
-rwxr-xr-x. 1 npg npg 35368 Mar  9  2021 merge64.new
-rwxrwxr-x. 1 npg npg 35910 Mar  8  2021 merge64.orig
-rwxr-xr-x. 1 npg npg 21370 Mar 16  2021 statrate64

./MergeServer/link:
total 164
-rw-rw-r--. 1 npg npg 10216 Oct 18  2019 handle_buffer.c
-rw-rw-r--. 1 npg npg 24623 Sep 19  2019 link_server.c
-rw-rw-r--. 1 npg npg 24732 Oct 18  2019 link_server.c~
drwxrwxr-x. 2 npg npg   217 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg  7051 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg  7029 Oct 18  2019 main.c~
-rw-rw-r--. 1 npg npg 14464 Sep 19  2019 processAIDADATAbuffer.c
-rw-rw-r--. 1 npg npg 14463 Oct 18  2019 processAIDADATAbuffer.c~
-rw-rw-r--. 1 npg npg 14950 Sep 19  2019 processR3BDATAbuffer.c
-rw-rw-r--. 1 npg npg 14925 Oct 18  2019 processR3BDATAbuffer.c~
-rw-rw-r--. 1 npg npg 15077 Aug 22  2019 processR3BDATAbuffer.c_PreAug22

./MergeServer/link/Linux64:
total 176
-rw-rw-r--. 1 npg npg  3088 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  6216 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 29840 Oct 24  2019 handle_buffer.o
-rwxrwxr-x. 1 npg npg 50660 Oct 24  2019 link64
-rw-rw-r--. 1 npg npg 24464 Oct 24  2019 link_server.o
-rw-rw-r--. 1 npg npg 14864 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   795 Aug 22  2019 Makefile
-rw-rw-r--. 1 npg npg   776 Apr 16  2019 Makefile~
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 12848 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/master:
total 40
-rw-rw-r--. 1 npg npg 7850 Sep 18  2019 handle_signals.c
drwxrwxr-x. 2 npg npg  223 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 6952 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 6973 Sep 19  2019 main.c~
-rw-rw-r--. 1 npg npg 1722 Aug 29  2019 start_link.c
-rw-rw-r--. 1 npg npg 1885 Aug 22  2019 start_link.c~
-rw-rw-r--. 1 npg npg 1372 Apr 16  2019 start_merge.c
-rw-rw-r--. 1 npg npg 1345 Apr 16  2019 start_statrate.c

./MergeServer/master/Linux64:
total 120
-rw-rw-r--. 1 npg npg  3408 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  4168 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 22840 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   793 Jul 16  2019 Makefile
-rwxrwxr-x. 1 npg npg 33184 Oct 24  2019 master64
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 14616 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3408 Oct 24  2019 start_link.o
-rw-rw-r--. 1 npg npg  2904 Oct 24  2019 start_merge.o
-rw-rw-r--. 1 npg npg  2808 Oct 24  2019 start_statrate.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/merge:
total 48
drwxrwxr-x. 2 npg npg  228 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 5130 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 4579 Oct 24  2019 mergeAIDA.c
-rw-rw-r--. 1 npg npg 4119 Oct 17  2019 #merge.c#
-rw-rw-r--. 1 npg npg 4148 Oct 21  2019 merge.c
-rw-rw-r--. 1 npg npg 3703 Sep 19  2019 mergeR3B.c
-rw-rw-r--. 1 npg npg 8060 Sep 17  2019 merge_server.c
-rw-rw-r--. 1 npg npg 2394 Sep 17  2019 storage.c

./MergeServer/merge/Linux64:
total 120
-rw-rw-r--. 1 npg npg  3088 Oct 24  2019 alloc_data_area.o
-rw-rw-r--. 1 npg npg  3872 Oct 24  2019 buffer.o
-rw-rw-r--. 1 npg npg 10944 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   789 Jul 16  2019 Makefile
-rw-rw-r--. 1 npg npg   776 Apr 16  2019 Makefile~
-rwxrwxr-x. 1 npg npg 35910 Oct 24  2019 merge64
-rw-rw-r--. 1 npg npg  9064 Oct 24  2019 merge.o
-rw-rw-r--. 1 npg npg  8160 Oct 24  2019 merge_server.o
-rw-rw-r--. 1 npg npg  5264 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 12744 Oct 24  2019 signal.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  4048 Oct 24  2019 storage.o
-rw-rw-r--. 1 npg npg  1864 Oct 24  2019 version.o

./MergeServer/statrate:
total 16
drwxrwxr-x. 2 npg npg  161 Oct 24  2019 Linux64
-rw-rw-r--. 1 npg npg 1876 Sep 19  2019 main.c
-rw-rw-r--. 1 npg npg 1717 Sep 19  2019 main.c~
-rw-rw-r--. 1 npg npg 2645 Apr 16  2019 statrate_server.c
-rw-rw-r--. 1 npg npg 2645 Apr 16  2019 statrate_server.c~

./MergeServer/statrate/Linux64:
total 80
-rw-rw-r--. 1 npg npg  8880 Oct 24  2019 main.o
-rw-rw-r--. 1 npg npg   725 Apr 16  2019 Makefile
-rw-rw-r--. 1 npg npg   728 Apr 16  2019 Makefile~
-rw-rw-r--. 1 npg npg  5376 Oct 24  2019 message.o
-rw-rw-r--. 1 npg npg 10144 Oct 24  2019 signal.o
-rwxrwxr-x. 1 npg npg 21370 Oct 24  2019 statrate64
-rw-rw-r--. 1 npg npg  4664 Oct 24  2019 statrate_server.o
-rw-rw-r--. 1 npg npg  3480 Oct 24  2019 stats.o
-rw-rw-r--. 1 npg npg  1872 Oct 24  2019 version.o

./tcl:
total 4
-rw-rw-r--. 1 npg npg 1775 Aug 26  2019 NewMergerControl.tcl
  630   Tue May 7 13:25:00 2024 JB, CC, NH, HMATuesday 6 May - dismount snout

Status before dismount

14:26 DSSSD#1 119.99 V @ 17.968 uA

          DSSSD#2 119.95 V @ 11.212 uA

          Leakage currents slightly elevated from Monday.

S4 temperature at 26 °C.

14:30 Detectors being debiased. All relay channels turned off. Voltages and currents of both DSSSDs to 0.

Plan:

- Dismount snout.

- Open up and remove BB7 and Upstream bPlast - leave AIDA 1 and 2 in place if possible.

 

 

Attachment 1: Screenshot_from_2024-05-07_14-25-52.png
Screenshot_from_2024-05-07_14-25-52.png
Attachment 2: Screenshot_from_2024-05-07_14-29-13.png
Screenshot_from_2024-05-07_14-29-13.png
  629   Mon May 6 07:10:26 2024 TDMonday 6 May
08.11 DSSSD#1 bias -120V leakage current -17.0uA
      DSSSD#2 bias -120V leakage current -10.5uA
      S4 temperature 25 deg C

      cf. 04.00 30.4.24 S4 temperature 25 deg C

      DSSSD#1 -21.9uA
      DSSSD#2 -13.0uA

      Leakage currents currently c. 80% of 30.4.24 values at same temperature 
Attachment 1: Capture.PNG
Capture.PNG
  628   Sun May 5 10:22:24 2024 TDSunday 5 May
11.22 DSSSD#1 bias -120V leakage current -17.1uA
      DSSSD#2 bias -120V leakage current -10.6uA
      S4 temperature 25 deg C

      cf. 07.00 29.4.24 S4 temperature 25 deg C

      DSSSD#1 -21.8uA
      DSSSD#2 -12.8uA

      Leakage currents currently c. 80% of 29.4.24 values at same temperature 
Attachment 1: Capture.PNG
Capture.PNG
  627   Sat May 4 07:36:07 2024 TDSaturday 4 May
08.32 DSSSD bias & leakage currents - Grafana - attachments 1-2
      FEE64 temperatures OK - attachment 3
      ADC, correlation scalers data item stats OK attachments 4-5
           11/16 <20k max aida16 174k
      per FEE64 Rate spectra - attachmnt 6


09.44 per p+n FEE64 1.8.L spectra - attachment 7
       aida09 pulser peak width 58 ch FWHM
      per FEE64 1.8.W spectra - attachments 8-10

09.47 bias cycle - off and then on
      no significant change in leakage current

09.57 FEE64 power OFF
      DSSSD bias ON
Attachment 1: Screenshot_from_2024-05-04_08-35-38.png
Screenshot_from_2024-05-04_08-35-38.png
Attachment 2: Screenshot_from_2024-05-04_08-33-29.png
Screenshot_from_2024-05-04_08-33-29.png
Attachment 3: Screenshot_from_2024-05-04_08-31-09.png
Screenshot_from_2024-05-04_08-31-09.png
Attachment 4: Screenshot_from_2024-05-04_08-31-45.png
Screenshot_from_2024-05-04_08-31-45.png
Attachment 5: Screenshot_from_2024-05-04_08-32-21.png
Screenshot_from_2024-05-04_08-32-21.png
Attachment 6: Screenshot_from_2024-05-04_08-32-41.png
Screenshot_from_2024-05-04_08-32-41.png
Attachment 7: Screenshot_from_2024-05-04_09-37-27.png
Screenshot_from_2024-05-04_09-37-27.png
Attachment 8: Screenshot_from_2024-05-04_09-42-27.png
Screenshot_from_2024-05-04_09-42-27.png
Attachment 9: Screenshot_from_2024-05-04_09-42-56.png
Screenshot_from_2024-05-04_09-42-56.png
Attachment 10: Screenshot_from_2024-05-04_09-44-31.png
Screenshot_from_2024-05-04_09-44-31.png
Attachment 11: Capture.PNG
Capture.PNG
  626   Thu May 2 23:08:43 2024 TDFriday 3 May
00.05 FEE64 temperatures OK
      ADC data item stats OK

09.31 FEE64 temperatures OK
      ADC data item stats OK

13.15 FEE64 temperatures OK
      ADC data item stats OK
       9/16 <20k max aida16 164k

20.15 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida03 no data - DAQ reset
       9/16 <20k max aida16 158k

23.15 FEE64 temperatures OK
      ADC data item stats
       aida03 no data - DAQ reset
       10/16 <20k max aida16 153k
  625   Thu May 2 00:16:22 2024 TD, NHThursday 2 May
01.15 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 no data
       6/16 < 20k


10.25 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 restarted
       7/16 < 20k, max 220k aida16

      all histograms zero'd

13.20 per p+n FEE64 1.8.L spectra - attachment 1
       aida09 pulser peak width 59 ch FWHM
      FEE64 temperatures OK
      ADC data item stats
       aida04 no data
       5/16 < 20k, max 332k aida08 

14.37
  Moved Time Machine delayed scaler from aida04 to aida07
  aida04 no data
  Reset merger etc 
  aida05 no data
  Reset merger etc 
  All FEEs with data again

  S4 Ambient Environment: 26.6 C / 30.4% RH / Td=7.8 C

17.26 FEE64 temperatures OK
      ADC data item stats OK

19.28 FEE64 temperatures OK
      ADC data item stats 
       aida02 no data - DAQ reset
      
Attachment 1: Screenshot_from_2024-05-02_13-21-07.png
Screenshot_from_2024-05-02_13-21-07.png
Attachment 2: Screenshot_from_2024-05-02_13-22-16.png
Screenshot_from_2024-05-02_13-22-16.png
Attachment 3: Screenshot_from_2024-05-02_13-22-43.png
Screenshot_from_2024-05-02_13-22-43.png
  624   Wed May 1 08:59:15 2024 TDWednesday 1 May
04.55 FEE64 temperatures OK
      ADC data item stats OK *except* aida04 no data
      TapeServer no storage mode

09.55 FEE64 temperatures OK
      ADC data item stats OK *except* aida02 & aida04 no data
      DSSSD bias and leakage current - Grafana - attachment 1

19.28 FEE64 temperatures OK
      ADC data item stats
       aida02 & aida04 no data
       noted increase in rates for aid08 and aida16
       5/16 < 20k
Attachment 1: Capture.PNG
Capture.PNG
  623   Tue Apr 30 09:06:45 2024 TD, NHTuesday 30 April
10.00 DSSSD bias & leakasge current - Grafana - attachments 1-2

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 4
       9/16 FEE64s < 20k 

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachments 6-7
       aida02, aida04 inactive

11.20 DAQ restarted
      aida02 and aida04 now producing data


11.30 analysis data file R22_313
      all deadtimes <1%
      no timewarps
      HEC data item rate 560Hz (all from one channel of aida08 asic #3)
      aida03 & aida04 MBS rates c. 24kHz


14.32 FEE64 temperatures OK
      ADC data item stats OK
       all FEE64s producing data

14.43 all histograms & stats zero'd

[NH checks here]

21.26 aida04 stopped forwarding to merger
 the "quick fix" (stop merger, rerun) worked

MBS is having huge issues on DESPEC side, AIDA not included
Decide to stop taking data to disk for now (no storage mode on)
Keeping running (incl. to AIDA FDR MBS) just to get an idea of stability

System Wide Checks
 clocks PASSED
 ADC IGNORED
 WR:
		 Base 		Current 	Difference
aida02 fault 	 0xe4d8 : 	 0x65c0 : 	 -32536  
aida03 fault 	 0x2371 : 	 0x2380 : 	 15  
aida04 fault 	 0xa210 : 	 0xaf58 : 	 3400  
aida06 fault 	 0x3624 : 	 0x3626 : 	 2  
White Rabbit error counter test result: Passed 12, Failed 4

FPGA: 
	
			 Base 		Current 		Difference
aida02 fault 	 0xaa3b : 	 0xc455 : 	 6682  
aida04 fault 	 0x5a4 : 	 0x6ba : 	 278  
aida06 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Module aida04 LMK PLL channel 1 lock transition counter has incremented
Module aida04 LMK PLL channel 2 lock transition counter has incremented
PLL lock counter test result: Passed 15, Failed 1

Baselines reset

22.50 FEE64 temps ok
      merger ok
       all links active
      ADC data item stats ok
Attachment 1: Screenshot_from_2024-04-30_10-05-08.png
Screenshot_from_2024-04-30_10-05-08.png
Attachment 2: Screenshot_from_2024-04-30_10-05-37.png
Screenshot_from_2024-04-30_10-05-37.png
Attachment 3: Screenshot_from_2024-04-30_10-01-32.png
Screenshot_from_2024-04-30_10-01-32.png
Attachment 4: Screenshot_from_2024-04-30_10-02-03.png
Screenshot_from_2024-04-30_10-02-03.png
Attachment 5: Screenshot_from_2024-04-30_10-03-26.png
Screenshot_from_2024-04-30_10-03-26.png
Attachment 6: Screenshot_from_2024-04-30_10-03-58.png
Screenshot_from_2024-04-30_10-03-58.png
Attachment 7: R22_313
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  246802750 (  787077.4 Hz)
        Other data format:   15117251 (   48210.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        802 (       2.6 Hz)
                               RESUME:        802 (       2.6 Hz)
                              SYNC100:      33168 (     105.8 Hz)
                              WR48-63:      33168 (     105.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   15049311 (   47993.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     182597 (     582.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      313.569 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.410                0.000
  4                0.000                0.000
  5                0.208                0.000
  6                0.000                0.000
  7                1.577                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                1.890                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                1.784                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4403022      10517          0          0          0          0        556        556          0       9405          0          0
  1    4797622      11663          0          0          0          0        607        607          0      10449          0          0
  2    9960742    7551016          0          0          0          0       2246       2246          0    7546524          0          0
  3   26989916    7491649          0          0         64         64       4294       4294          0    7482933          0          0
  4    2737535        758          0          0          0          0        379        379          0          0          0          0
  5   21805196       5662          0          0         70         70       2761       2761          0          0          0          0
  6    6978804       1886          0          0          0          0        943        943          0          0          0          0
  7   53329550      13820          0          0        126        126       6784       6784          0          0          0     182597
  8     479454        122          0          0          0          0         61         61          0          0          0          0
  9    5592721       1538          0          0          0          0        769        769          0          0          0          0
 10   39929583      11138          0          0        441        441       5128       5128          0          0          0          0
 11    2351324        636          0          0          0          0        318        318          0          0          0          0
 12    5309169       1226          0          0          0          0        613        613          0          0          0          0
 13    3913898       1006          0          0          0          0        503        503          0          0          0          0
 14    2364855        578          0          0          0          0        289        289          0          0          0          0
 15   55859359      14036          0          0        101        101       6917       6917          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.668s (  979.553 blocks/s,  61.222 Mb/s)
  622   Mon Apr 29 08:11:16 2024 TDMonday 29 April
08.52 Beam off

09.11 New data file R22
      Monitor accumulated implanted activity in bPlas and DSSSDs

      ASIC settings 2024Apr26-01.29.53

      DSSSD bias & leakage current OK - attachments 1-3

      FEE64 temperatures OK - attachment 4

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 6

      Merger etc - attachment 7

      analysis data file R22_19 - attachment 8
       all deadtimes 0.3%, or less
       no timewarps
       HEC data item rate 390Hz

   
10.00 all histograms & stats zero'd


13.24 DSSSD bias & leakage current OK - attachments 9-10

      FEE64 temperatures OK - attachment 11

      ADC data item stats - attachment 12

      per FEE64 Rate spectra - attachment 13

      per p+n FEE64 1.8.L spectra - attachments 14-16
       common x/y axes - peak height proxy for width
       aida09 pulser peak width 58 ch FWHM (~41keV FWHM)  cf. https://elog.ph.ed.ac.uk/DESPEC/585 attachment 38
       electronic noise today very similar to electronic noise Monday 22 April

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-19

      Merger etc - attachment 20

      analysis data file R22_60 - attachment 21
       max deadtime 1.7% (aida11), all others <1% 
       no timewarps
       HEC data item rate 710Hz

14.13 ucesb - attachment 22

14.15 aida04 not producing data

14.33 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618

15.27 *all* FEE64s stop producing data - reason unknown

16.15 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618


16.29 DSSSD bias & leakage current OK - attachments 23-24

      FEE64 temperatures OK - attachment 25

      ADC data item stats - attachment 26

      per FEE64 Rate spectra - attachment 27

      Merger etc - attachment 28


20.45 DSSSD bias & leakage current OK - attachments 29-30

      FEE64 temperatures OK - attachment 32

      ADC data item stats - attachment 31

      per FEE64 Rate spectra - attachment 33

      Merger etc - attachment 34

20.54  analysis data file R22_145 - attachment 35
       max deadtime 1.5% (aida11), all others <1% 
       no timewarps
       HEC data item rate 775Hz



03.45 DSSSD bias & leakage current OK - attachments 36-37

      FEE64 temperatures OK - attachment 38

      ADC data item stats - attachment 39

      per FEE64 Rate spectra - attachment 40

      Merger etc - attachment 41
Attachment 1: Screenshot_from_2024-04-29_09-13-12.png
Screenshot_from_2024-04-29_09-13-12.png
Attachment 2: Screenshot_from_2024-04-29_09-13-37.png
Screenshot_from_2024-04-29_09-13-37.png
Attachment 3: Screenshot_from_2024-04-29_09-13-56.png
Screenshot_from_2024-04-29_09-13-56.png
Attachment 4: Screenshot_from_2024-04-29_09-14-31.png
Screenshot_from_2024-04-29_09-14-31.png
Attachment 5: Screenshot_from_2024-04-29_09-15-05.png
Screenshot_from_2024-04-29_09-15-05.png
Attachment 6: Screenshot_from_2024-04-29_09-15-48.png
Screenshot_from_2024-04-29_09-15-48.png
Attachment 7: Screenshot_from_2024-04-29_09-17-25.png
Screenshot_from_2024-04-29_09-17-25.png
Attachment 8: R22_19
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  245162657 (  661369.3 Hz)
        Other data format:   16757344 (   45205.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        432 (       1.2 Hz)
                               RESUME:        432 (       1.2 Hz)
                              SYNC100:      33381 (      90.1 Hz)
                              WR48-63:      33381 (      90.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   16689718 (   45023.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     145285 (     391.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      370.689 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.421                0.000
  4                0.000                0.000
  5                0.062                0.000
  6                0.000                0.000
  7                1.097                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                1.032                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                1.195                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4689453      12333          0          0          0          0        606        606          0      11121          0        151
  1    6714412    1102163          0          0          0          0        995        995          0    1100173          0        247
  2    9093299    6196435          0          0          0          0       2024       2024          0    6192387          0        151
  3   31978220    6022790          0          0         67         67       4731       4731          0    6013194          0        212
  4    3345145    1695016          0          0          0          0        641        641          0    1693734          0         75
  5   18560198    1684371          0          0         12         12       2619       2619          0    1679109          0        279
  6    7471590       2034          0          0          0          0       1017       1017          0          0          0        150
  7   51047482      13240          0          0        105        105       6515       6515          0          0          0     143158
  8    1099097        280          0          0          0          0        140        140          0          0          0        152
  9    7056038       1776          0          0          0          0        888        888          0          0          0        136
 10   35026945       9268          0          0        144        144       4490       4490          0          0          0         97
 11    2488733        676          0          0          0          0        338        338          0          0          0        118
 12    6088987       1632          0          0          0          0        816        816          0          0          0         53
 13    5057582       1278          0          0          0          0        639        639          0          0          0        108
 14    2541330        642          0          0          0          0        321        321          0          0          0        127
 15   52904146      13410          0          0        104        104       6601       6601          0          0          0         71
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.137s (  965.696 blocks/s,  60.356 Mb/s)
Attachment 9: Screenshot_from_2024-04-29_13-24-45.png
Screenshot_from_2024-04-29_13-24-45.png
Attachment 10: Screenshot_from_2024-04-29_13-25-48.png
Screenshot_from_2024-04-29_13-25-48.png
Attachment 11: Screenshot_from_2024-04-29_13-26-19.png
Screenshot_from_2024-04-29_13-26-19.png
Attachment 12: Screenshot_from_2024-04-29_13-26-51.png
Screenshot_from_2024-04-29_13-26-51.png
Attachment 13: Screenshot_from_2024-04-29_13-27-33.png
Screenshot_from_2024-04-29_13-27-33.png
Attachment 14: Screenshot_from_2024-04-29_13-28-45.png
Screenshot_from_2024-04-29_13-28-45.png
Attachment 15: Screenshot_from_2024-04-29_13-29-15.png
Screenshot_from_2024-04-29_13-29-15.png
Attachment 16: Screenshot_from_2024-04-29_13-30-17.png
Screenshot_from_2024-04-29_13-30-17.png
Attachment 17: Screenshot_from_2024-04-29_13-32-51.png
Screenshot_from_2024-04-29_13-32-51.png
Attachment 18: Screenshot_from_2024-04-29_13-33-20.png
Screenshot_from_2024-04-29_13-33-20.png
Attachment 19: Screenshot_from_2024-04-29_13-34-07.png
Screenshot_from_2024-04-29_13-34-07.png
Attachment 20: Screenshot_from_2024-04-29_13-34-32.png
Screenshot_from_2024-04-29_13-34-32.png
Attachment 21: R22_60
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258414198 (  779475.1 Hz)
        Other data format:    3505804 (   10574.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        880 (       2.7 Hz)
                               RESUME:        880 (       2.7 Hz)
                              SYNC100:      33235 (     100.2 Hz)
                              WR48-63:      33235 (     100.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3437574 (   10369.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     235960 (     711.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      331.523 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.095                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.217                0.000
  6                0.140                0.000
  7                2.432                0.000
  8                0.000                0.000
  9                0.011                0.000
 10                5.658                0.000
 11                0.000                0.000
 12                0.051                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                2.652                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4927110      11209          0          0          4          4        628        628          0       9945          0         39
  1    5556527     109296          0          0          0          0        741        741          0     107814          0         59
  2   10335604    3081300          0          0          0          0       1668       1668          0    3077964          0         29
  3          0          0          0          0          0          0          0          0          0          0          0          0
  4    3176904     121984          0          0          0          0        428        428          0     121128          0         19
  5   28235398     128309          0          0         73         73       3720       3720          0     120723          0         52
  6    7534053       1958          0          0          2          2        977        977          0          0          0         22
  7   65092755      17088          0          0        173        173       8371       8371          0          0          0     235567
  8     631770        138          0          0          0          0         69         69          0          0          0         28
  9    6806363       1686          0          0          1          1        842        842          0          0          0         28
 10   44391302      12266          0          0        505        505       5628       5628          0          0          0         16
 11    2632024        670          0          0          0          0        335        335          0          0          0         21
 12    5950079       1370          0          0          1          1        684        684          0          0          0         14
 13    4496081       1136          0          0          0          0        568        568          0          0          0         29
 14    2644608        652          0          0          0          0        326        326          0          0          0         25
 15   66003620      16742          0          0        121        121       8250       8250          0          0          0         12
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   37.793s (  846.718 blocks/s,  52.920 Mb/s)
Attachment 22: Capture.PNG
Capture.PNG
Attachment 23: Screenshot_from_2024-04-29_16-29-34.png
Screenshot_from_2024-04-29_16-29-34.png
Attachment 24: Screenshot_from_2024-04-29_16-29-54.png
Screenshot_from_2024-04-29_16-29-54.png
Attachment 25: Screenshot_from_2024-04-29_16-30-39.png
Screenshot_from_2024-04-29_16-30-39.png
Attachment 26: Screenshot_from_2024-04-29_16-31-01.png
Screenshot_from_2024-04-29_16-31-01.png
Attachment 27: Screenshot_from_2024-04-29_16-31-51.png
Screenshot_from_2024-04-29_16-31-51.png
Attachment 28: Screenshot_from_2024-04-29_16-32-12.png
Screenshot_from_2024-04-29_16-32-12.png
Attachment 29: Screenshot_from_2024-04-29_20-45-26.png
Screenshot_from_2024-04-29_20-45-26.png
Attachment 30: Screenshot_from_2024-04-29_20-46-25.png
Screenshot_from_2024-04-29_20-46-25.png
Attachment 31: Screenshot_from_2024-04-29_20-46-52.png
Screenshot_from_2024-04-29_20-46-52.png
Attachment 32: Screenshot_from_2024-04-29_20-47-20.png
Screenshot_from_2024-04-29_20-47-20.png
Attachment 33: Screenshot_from_2024-04-29_20-48-08.png
Screenshot_from_2024-04-29_20-48-08.png
Attachment 34: Screenshot_from_2024-04-29_20-48-56.png
Screenshot_from_2024-04-29_20-48-56.png
Attachment 35: R22_145
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  247656598 (  992529.4 Hz)
        Other data format:   14263402 (   57163.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        881 (       3.5 Hz)
                               RESUME:        881 (       3.5 Hz)
                              SYNC100:      32930 (     132.0 Hz)
                              WR48-63:      32930 (     132.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   14195780 (   56892.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     193328 (     774.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      249.521 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.003                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.318                0.000
  4                0.000                0.000
  5                1.537                0.000
  6                0.000                0.000
  7                1.906                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                3.841                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                1.886                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6683029       9310          0          0          5          5        906        906          0       7488          0         74
  1    5367157    1313019          0          0          0          0        827        827          0    1311365          0         63
  2   11584446    4358913          0          0          0          0       1980       1980          0    4354953          0         37
  3   22834625    4167004          0          0         37         37       3370       3370          0    4160190          0         57
  4    3277605    2130735          0          0          0          0        657        657          0    2129421          0         19
  5   24294581    2239601          0          0        194        194       3425       3425          0    2232363          0         49
  6    6724993       1760          0          0          0          0        880        880          0          0          0         28
  7   50507044      12910          0          0        126        126       6329       6329          0          0          0     192806
  8     656116        188          0          0          0          0         94         94          0          0          0         10
  9    5715462       1410          0          0          0          0        705        705          0          0          0         12
 10   39787841      10690          0          0        431        431       4914       4914          0          0          0          4
 11    3128950        784          0          0          0          0        392        392          0          0          0         67
 12    4910404       1244          0          0          0          0        622        622          0          0          0         13
 13    3786534        940          0          0          0          0        470        470          0          0          0         59
 14    2967123        782          0          0          0          0        391        391          0          0          0          2
 15   55430688      14112          0          0         88         88       6968       6968          0          0          0         28
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.727s (  948.807 blocks/s,  59.300 Mb/s)
Attachment 36: Screenshot_from_2024-04-30_03-45-13.png
Screenshot_from_2024-04-30_03-45-13.png
Attachment 37: Screenshot_from_2024-04-30_03-46-07.png
Screenshot_from_2024-04-30_03-46-07.png
Attachment 38: Screenshot_from_2024-04-30_03-46-43.png
Screenshot_from_2024-04-30_03-46-43.png
Attachment 39: Screenshot_from_2024-04-30_03-47-10.png
Screenshot_from_2024-04-30_03-47-10.png
Attachment 40: Screenshot_from_2024-04-30_03-47-40.png
Screenshot_from_2024-04-30_03-47-40.png
Attachment 41: Screenshot_from_2024-04-30_03-48-26.png
Screenshot_from_2024-04-30_03-48-26.png
  621   Sun Apr 28 22:52:06 2024 Marc0:00-08:00 Monday 29 April

Just another manic magic Monday, ....

Taking over Betool's AIDA shift. All is good.

This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.

01:15: AIDA04 is down again. DAQ shift crew informed.

01:27: AIDA04 is back and shift crew has been informed.

01:30 Checks:

     Bias and leakage current - Attachment 1-2

     Temperatures - Attachment 3

     ADC Data item - Attachment 4

     Rate Spectra - Attachment 5

     UCESB - Attachment  6

     Timestamp checks - Attachement 7

06:00 It has run very smoothly.   

 

Attachment 1: Screenshot_from_2024-04-29_01-33-27.png
Screenshot_from_2024-04-29_01-33-27.png
Attachment 2: Screenshot_from_2024-04-29_01-34-37.png
Screenshot_from_2024-04-29_01-34-37.png
Attachment 3: Screenshot_from_2024-04-29_01-36-12.png
Screenshot_from_2024-04-29_01-36-12.png
Attachment 4: Screenshot_from_2024-04-29_01-36-41.png
Screenshot_from_2024-04-29_01-36-41.png
Attachment 5: Screenshot_from_2024-04-29_01-37-56.png
Screenshot_from_2024-04-29_01-37-56.png
Attachment 6: Screenshot_from_2024-04-29_01-32-39.png
Screenshot_from_2024-04-29_01-32-39.png
Attachment 7: Screenshot_from_2024-04-29_01-42-58.png
Screenshot_from_2024-04-29_01-42-58.png
  620   Sun Apr 28 20:35:57 2024 Betool Alayed16:0-00:00 28 April 2024

21:30 screenshots

and

23:24 screenshots

Attachment 1: Screenshot_from_2024-04-28_21-30-19.png
Screenshot_from_2024-04-28_21-30-19.png
Attachment 2: Screenshot_from_2024-04-28_21-31-11.png
Screenshot_from_2024-04-28_21-31-11.png
Attachment 3: Screenshot_from_2024-04-28_21-32-07.png
Screenshot_from_2024-04-28_21-32-07.png
Attachment 4: Screenshot_from_2024-04-28_21-32-51.png
Screenshot_from_2024-04-28_21-32-51.png
Attachment 5: Screenshot_from_2024-04-28_21-35-10.png
Screenshot_from_2024-04-28_21-35-10.png
Attachment 6: Screenshot_from_2024-04-28_23-35-25.png
Screenshot_from_2024-04-28_23-35-25.png
Attachment 7: Screenshot_from_2024-04-28_23-34-31.png
Screenshot_from_2024-04-28_23-34-31.png
Attachment 8: Screenshot_from_2024-04-28_23-33-42.png
Screenshot_from_2024-04-28_23-33-42.png
Attachment 9: Screenshot_from_2024-04-28_23-32-28.png
Screenshot_from_2024-04-28_23-32-28.png
Attachment 10: Screenshot_from_2024-04-28_23-31-51.png
Screenshot_from_2024-04-28_23-31-51.png
  619   Sun Apr 28 16:26:47 2024 Betool Alayed16:00-00:00 Sunday 28 April

19:24 pm screenshots

and

17:18 pm screenshots

Attachment 1: Screenshot_from_2024-04-28_17-18-38.png
Screenshot_from_2024-04-28_17-18-38.png
Attachment 2: Screenshot_from_2024-04-28_17-13-13.png
Screenshot_from_2024-04-28_17-13-13.png
Attachment 3: Screenshot_from_2024-04-28_17-11-48.png
Screenshot_from_2024-04-28_17-11-48.png
Attachment 4: Screenshot_from_2024-04-28_17-10-32.png
Screenshot_from_2024-04-28_17-10-32.png
Attachment 5: Screenshot_from_2024-04-28_17-04-10.png
Screenshot_from_2024-04-28_17-04-10.png
Attachment 6: Screenshot_from_2024-04-28_19-18-42.png
Screenshot_from_2024-04-28_19-18-42.png
Attachment 7: Screenshot_from_2024-04-28_19-20-10.png
Screenshot_from_2024-04-28_19-20-10.png
Attachment 8: Screenshot_from_2024-04-28_19-22-19.png
Screenshot_from_2024-04-28_19-22-19.png
Attachment 9: Screenshot_from_2024-04-28_19-23-19.png
Screenshot_from_2024-04-28_19-23-19.png
Attachment 10: Screenshot_from_2024-04-28_19-24-40.png
Screenshot_from_2024-04-28_19-24-40.png
  618   Sun Apr 28 16:07:59 2024 TD[How To] Recover DAQ when one FEE64 stops sending data - short(er) version
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5)   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

6) Start DAQ

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

8) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

9) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

10) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.
  617   Sun Apr 28 15:49:58 2024 TDDAQ restart short(er) cut
16.45 aida04 stopped producing data

      DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)

       aida04 did not stop, all other FEE64s stopped

      restart Merger (workspace 3, display 2, select 'NewMerger' icon)

      SETUP Merger (workspace 3, display 1, browser 'NewMerger' tab)
      GO Merger (workspace 3, display 1, browser 'NewMerger' tab)

       aida04 now stops

      disable *all* #1 data transfer 

      DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
      check WR timestamps OK (workspace 2, display 1, GSI WR Timestamp Control)

      DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
      enable *all* #1 data transfer
      DAQ GO (workspace 2, display 1, browser 'Run Control' tab)


      No reboot or power cycle required. Just clear the Merger data links, restart DAQ.
      
  616   Sun Apr 28 15:09:34 2024 Betool Alayed16:0-00:00 28 April 2024

16:00 aida04 not producing data

          power cycle all FEE64s to recover DAQ

16.20 analysis data file R21_668

         max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%

         no timewarps

         HEC data item rate 1.9kHz

 

Attachment 1: R21_668.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256513573 ( 1160512.7 Hz)
        Other data format:    5406427 (   24459.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1079 (       4.9 Hz)
                               RESUME:       1078 (       4.9 Hz)
                              SYNC100:      32824 (     148.5 Hz)
                              WR48-63:      32824 (     148.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5338622 (   24152.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     440937 (    1994.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      221.035 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.085                0.000
  1               19.736                0.000
  2                0.053                0.000
  3               36.607                0.000
  4                0.854                0.000
  5                4.747                0.000
  6                0.000                0.000
  7                1.845                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.661                0.000
 11                0.020                0.000
 12                0.004                0.000
 13                0.000                0.000
 14                0.003                0.000
 15                1.247                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8138410       8624          0          0         12         12        985        985          0       6630          0      38979
  1   27142829     245404          0          0        242        242       3461       3461          0     237998          0      64179
  2   11096772    2350080          0          0          8          8       1768       1768          0    2346528          0      26862
  3   37888832    1835283          0          0        304        303       4762       4762          0    1825152          0      18340
  4    9329455     413600          0          0         28         28       1212       1212          0     411120          0      29227
  5   23846951     517930          0          0        167        167       3201       3201          0     511194          0      24385
  6    5635241       1420          0          0          0          0        710        710          0          0          0       8053
  7   40074524      10318          0          0         93         93       5066       5066          0          0          0     120554
  8    3088839        754          0          0          0          0        377        377          0          0          0      14672
  9    5102920       1254          0          0          0          0        627        627          0          0          0       6080
 10   25963054       6756          0          0        140        140       3238       3238          0          0          0       2640
 11    5980678       1458          0          0          5          5        724        724          0          0          0      52714
 12    5769255       1548          0          0          2          2        772        772          0          0          0       8398
 13    3930590        926          0          0          0          0        463        463          0          0          0       9544
 14    5399163       1328          0          0          1          1        663        663          0          0          0       5692
 15   38126060       9744          0          0         77         77       4795       4795          0          0          0      10618
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.004s (  941.068 blocks/s,  58.817 Mb/s)
  615   Sun Apr 28 06:57:59 2024 Magda Satrazani08:00-16:00 Sunday 28 April

Took over the shift from Marc.

 

 

10.51 Added screenshots to illustrate erroneous data from Grafana (AIDA) - attachments 18-19

          Compare to the CAEN HV control terminal (attachment 15) which shows the expected values. Have email'd Nic & Helena. TD

10.55 Noticed that the CAEN HV control terminal is showing the corerct data but on the wrong line of the display (attachment 15). Enter commands q (quit) and d (display) to restart display - the correct data now appears on the correct line (attachment 20).

         Grafana (AIDA) is now reporting data correctly too (attachment 21).

          Summary - CAEN HV terminal display issue ( c. 10.05 - 10.55 )

                             Note the correct data/values continued to be shown. Leakage current shown correctly by both Grafana and CAEN HV control throughout.

                             Conclude this was a display issue - not a change in the operating parameters of the CAEN N1419ET.. Data should be OK.

 

14.45 analysis data file R21_500 - attachment 36

         max deadtime 22% (aida11), 16% (aida04), 7% (aida02), 6% (aida08), 6% (aida06) , all others < 1%

         no timewarps

         HEC data item rate 2.7kHz

 

Attachment 1: Screenshot_2024-04-28_at_08-00-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-04-28_at_08-00-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
Attachment 3: leakage_current.png
leakage_current.png
Attachment 4: Screenshot_2024-04-28_at_08-05-38_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-04-28_at_08-05-38_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2024-04-28_at_08-08-14_ucesb.png
Screenshot_2024-04-28_at_08-08-14_ucesb.png
Attachment 6: Screenshot_2024-04-28_at_08-09-21_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_08-09-21_AIDA_-_Grafana.png
Attachment 7: Screenshot_2024-04-28_at_08-11-20_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_08-11-20_AIDA_Alerting_-_Grafana.png
Attachment 8: Screenshot_2024-04-28_at_10-01-48_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-04-28_at_10-01-48_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
Attachment 10: Screenshot_2024-04-28_at_10-02-44_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-04-28_at_10-02-44_Spectrum_Browser_aidas-gsi.png
Attachment 11: Screenshot_2024-04-28_at_10-03-40_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_10-03-40_AIDA_-_Grafana.png
Attachment 12: Screenshot_2024-04-28_at_10-04-04_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_10-04-04_AIDA_Alerting_-_Grafana.png
Attachment 13: Screenshot_2024-04-28_at_10-04-19_ucesb.png
Screenshot_2024-04-28_at_10-04-19_ucesb.png
Attachment 14: Screenshot_from_2024-04-28_10-08-23.png
Screenshot_from_2024-04-28_10-08-23.png
Attachment 15: Screenshot_from_2024-04-28_10-30-24.png
Screenshot_from_2024-04-28_10-30-24.png
Attachment 16: Screenshot_2024-04-28_at_10-34-47_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_10-34-47_AIDA_-_Grafana.png
Attachment 17: Screenshot_2024-04-28_at_10-34-56_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_10-34-56_AIDA_Alerting_-_Grafana.png
Attachment 18: Capture_Grafana.PNG
Capture_Grafana.PNG
Attachment 19: Capture_Grafana_2.PNG
Capture_Grafana_2.PNG
Attachment 20: Screenshot_from_2024-04-28_10-58-43.png
Screenshot_from_2024-04-28_10-58-43.png
Attachment 21: Capture_Grafana_3.PNG
Capture_Grafana_3.PNG
Attachment 22: Screenshot_from_2024-04-28_12-00-49.png
Screenshot_from_2024-04-28_12-00-49.png
Attachment 23: Screenshot_2024-04-28_at_12-01-21_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-04-28_at_12-01-21_Temperature_and_status_scan_aidas-gsi.png
Attachment 24: Screenshot_2024-04-28_at_12-02-58_Statistics_aidas-gsi.png
Screenshot_2024-04-28_at_12-02-58_Statistics_aidas-gsi.png
Attachment 25: Screenshot_2024-04-28_at_12-02-36_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-04-28_at_12-02-36_Spectrum_Browser_aidas-gsi.png
Attachment 26: Screenshot_2024-04-28_at_11-59-21_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_11-59-21_AIDA_-_Grafana.png
Attachment 27: Screenshot_2024-04-28_at_11-59-44_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_11-59-44_AIDA_Alerting_-_Grafana.png
Attachment 28: Screenshot_2024-04-28_at_12-00-08_ucesb.png
Screenshot_2024-04-28_at_12-00-08_ucesb.png
Attachment 29: Screenshot_2024-04-28_at_14-03-06_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-04-28_at_14-03-06_Temperature_and_status_scan_aidas-gsi.png
Attachment 30: Screenshot_2024-04-28_at_14-04-21_Statistics_aidas-gsi.png
Screenshot_2024-04-28_at_14-04-21_Statistics_aidas-gsi.png
Attachment 31: Screenshot_2024-04-28_at_14-04-07_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-04-28_at_14-04-07_Spectrum_Browser_aidas-gsi.png
Attachment 32: Screenshot_2024-04-28_at_14-01-38_AIDA_-_Grafana.png
Screenshot_2024-04-28_at_14-01-38_AIDA_-_Grafana.png
Attachment 33: Screenshot_2024-04-28_at_14-01-52_AIDA_Alerting_-_Grafana.png
Screenshot_2024-04-28_at_14-01-52_AIDA_Alerting_-_Grafana.png
Attachment 34: Screenshot_2024-04-28_at_14-02-02_ucesb.png
Screenshot_2024-04-28_at_14-02-02_ucesb.png
Attachment 35: Screenshot_from_2024-04-28_14-02-43.png
Screenshot_from_2024-04-28_14-02-43.png
Attachment 36: R21_500.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259329603 ( 2382071.3 Hz)
        Other data format:    2590397 (   23794.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1783 (      16.4 Hz)
                               RESUME:       1783 (      16.4 Hz)
                              SYNC100:      32405 (     297.7 Hz)
                              WR48-63:      32405 (     297.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2522021 (   23166.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     289080 (    2655.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      108.867 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                1.124                0.000
  1                7.512                0.000
  2                0.941                0.000
  3               17.003                0.000
  4                0.495                0.000
  5                6.767                0.000
  6                0.004                0.000
  7                7.595                0.000
  8                0.000                0.000
  9                0.006                0.000
 10               24.536                0.000
 11                0.145                0.000
 12                0.001                0.000
 13                0.000                0.000
 14                0.322                0.000
 15                8.584                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22169110       8971          0          0         57         57       2822       2822          0       3213          0      16255
  1   12639517     112315          0          0         99         99       1612       1612          0     108893          0      29028
  2   22281064    1127033          0          0         37         37       2985       2985          0    1120989          0      11197
  3   20798124     911187          0          0        135        135       2715       2715          0     905487          0       8456
  4    9363935     176936          0          0         14         14       1205       1205          0     174498          0      11991
  5   27019969     216459          0          0        325        325       3434       3434          0     208941          0       9910
  6    6076361       1558          0          0          2          2        777        777          0          0          0       3470
  7   35565405       8998          0          0        189        189       4310       4310          0          0          0     153479
  8    1477425        362          0          0          0          0        181        181          0          0          0       6096
  9    6250272       1506          0          0          2          2        751        751          0          0          0       2572
 10   27290515       8024          0          0        662        662       3350       3350          0          0          0        840
 11   10865846       2620          0          0         16         16       1294       1294          0          0          0      22001
 12    3883231        948          0          0          2          2        472        472          0          0          0       3485
 13    3455565        826          0          0          0          0        413        413          0          0          0       4071
 14   11216035       2794          0          0         51         51       1346       1346          0          0          0       2348
 15   38977229       9860          0          0        192        192       4738       4738          0          0          0       3881
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   38.605s (  828.898 blocks/s,  51.806 Mb/s)
  614   Sat Apr 27 23:04:42 2024 Marc00:00-08:00 Sunday 28 April

Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C

0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.

After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.

I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.

I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.

We are now back in business with AIDA back in main DAQ at  01:40

01:45 full checks:

           Bias: AIDA-Graphana - Attachment 1

               Leakage current ramping up again ! S4 temperaturenot really going down either: 25C

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 5

      ucesb - attachment 6

03:00 full checks

          No change to be highlighted. All pretty much the same as during previous full checks.

05:00 full checks

Expt still running very smoothly.

           Bias: AIDA-Graphana - Attachment 7-8

               Leakage current going down slightly

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
      per FEE64 Rate spectra - attachment 11

      Merger etc - attachment 12

      ucesb - attachment 13

07:15 all running smoothly.

07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.

07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.

 

Attachment 1: Screenshot_from_2024-04-28_01-51-37.png
Screenshot_from_2024-04-28_01-51-37.png
Attachment 2: Screenshot_from_2024-04-28_01-53-29.png
Screenshot_from_2024-04-28_01-53-29.png
Attachment 3: Screenshot_from_2024-04-28_01-53-52.png
Screenshot_from_2024-04-28_01-53-52.png
Attachment 4: Screenshot_from_2024-04-28_01-54-16.png
Screenshot_from_2024-04-28_01-54-16.png
Attachment 5: Screenshot_from_2024-04-28_01-56-14.png
Screenshot_from_2024-04-28_01-56-14.png
Attachment 6: Screenshot_from_2024-04-28_04-59-46.png
Screenshot_from_2024-04-28_04-59-46.png
Attachment 7: Screenshot_from_2024-04-28_04-59-51.png
Screenshot_from_2024-04-28_04-59-51.png
Attachment 8: Screenshot_from_2024-04-28_05-00-33.png
Screenshot_from_2024-04-28_05-00-33.png
Attachment 9: Screenshot_from_2024-04-28_05-00-25.png
Screenshot_from_2024-04-28_05-00-25.png
Attachment 10: Screenshot_from_2024-04-28_05-00-38.png
Screenshot_from_2024-04-28_05-00-38.png
Attachment 11: Screenshot_from_2024-04-28_05-02-44.png
Screenshot_from_2024-04-28_05-02-44.png
Attachment 12: Screenshot_from_2024-04-28_05-00-03.png
Screenshot_from_2024-04-28_05-00-03.png
  613   Sat Apr 27 15:13:06 2024 Norah16:00 - 00:00 Saturday 27 April

14:50 Checks

Leakage current  is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1

It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.

18:15 Checks

Everything appears to be going smoothly 

Grafana - DSSSD bias  and leakage current - attachments 2 and 3

FEE64 temperatures  - It appears okay, nothing strange , attachment 4

ADC data item stats - attachment 5

Per FEE64 Rate spectra - attachment 6

 ucesb - attachment 7

Merger Link Data Rates  - attachment 8

 

18:16

The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.

18:20  it came back to work normally

............................

20:00 Checks

Nothing new to report. All is well.

20:33 Aida02 showed as not reading - attachment 10

20:34 it resumed reading and returned to normal operation.

21:08 Aida02 was  not reading - attachment 11 , had to emailed Tom to fix it .

The shift crew have restarted ucesb and AIDA  is now showing OK - attachment 12

so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.

ADC data item stats and Merger etc  appear to be going smoothly - attachments 13 and 14

...........................................

22:00 Checks

Still running smoothly.

DSSD bias and  leakage current - ok

FEE64 temperatures - ok

Per FEE64 Rate spectra - ok

ADC data item stats - ok

Ucesb - ok

.................................

23:00 Checks

 Nothing new to report. Everything is going well , and I have captured screenshots of each one

Leakage currents dropped a bit See attachment 15

.............................

23:50 Checks

All seems smooth.

Attachment 1: Screenshot_from_2024-04-27_16-54-13.png
Screenshot_from_2024-04-27_16-54-13.png
Attachment 2: Screenshot_from_2024-04-27_18-15-47.png
Screenshot_from_2024-04-27_18-15-47.png
Attachment 3: Screenshot_from_2024-04-27_18-09-13.png
Screenshot_from_2024-04-27_18-09-13.png
Attachment 4: Screenshot_from_2024-04-27_18-07-50.png
Screenshot_from_2024-04-27_18-07-50.png
Attachment 5: Screenshot_from_2024-04-27_18-06-46.png
Screenshot_from_2024-04-27_18-06-46.png
Attachment 6: Screenshot_from_2024-04-27_18-07-19.png
Screenshot_from_2024-04-27_18-07-19.png
Attachment 7: Screenshot_from_2024-04-27_18-08-53.png
Screenshot_from_2024-04-27_18-08-53.png
Attachment 8: Screenshot_from_2024-04-27_18-09-57.png
Screenshot_from_2024-04-27_18-09-57.png
Attachment 9: Screenshot_from_2024-04-27_19-18-03.png
Screenshot_from_2024-04-27_19-18-03.png
Attachment 10: Screenshot_from_2024-04-27_20-33-01.png
Screenshot_from_2024-04-27_20-33-01.png
Attachment 11: Screenshot_from_2024-04-27_20-52-28.png
Screenshot_from_2024-04-27_20-52-28.png
Attachment 12: Screenshot_from_2024-04-27_21-22-51.png
Screenshot_from_2024-04-27_21-22-51.png
Attachment 13: Screenshot_from_2024-04-27_21-44-02.png
Screenshot_from_2024-04-27_21-44-02.png
Attachment 14: Screenshot_from_2024-04-27_21-45-02.png
Screenshot_from_2024-04-27_21-45-02.png
Attachment 15: Screenshot_from_2024-04-27_23-26-06.png
Screenshot_from_2024-04-27_23-26-06.png
  612   Sat Apr 27 04:59:31 2024 TD08:00-16:00 Saturday 27 April
06.00 DAQ continues file S100_alpha/R20_740
      FRS setting 162Eu

      DSSSD bias & leakage current - Grafana - attachments 1-2
       leakage currents tracking S4 ambient temperature

      FEE64 temperatures OK - attachment 3

      All system wide checks OK *except* 
       as reported - attachments 4-5
       *all* ADCs not calibrated (not calibrated post-synchronise ASIC clocks at most recent restart)

      WR timestamps OK - attachment 6

      ADC, PAUSE, RESUME, Correlation scaler data item stats OK - attachments 7-10

      per FEE64 Rate spectra - attachment 11

      per p+n FEE64 1.8.L specra - attachments 12-13
       aida09 pulser peak width 55 ch FWHM (~38keV FWHM)

      per FEE64 1.8.H spectra - attachments 14-15

      Merger, TapeServer etc - attachments 16-17
       12Mb/s to disk

      ucesb - attachment 18
       spill length 3.5s, extraction 1.5s

06.30 analysis data file R20_748 - attachment 19
      max deadtime 12%
      no timewarps
      HEC data item rate 2.1kHz

06.35 new run file S100_alpha/R21

09.30 DAQ continues - file S100_alpha/R21_83

      DSSSD bias & leakage currents - Grafana - attachment 20

      FEE64 temperatures OK - attachment 21

      ADC data item stats OK - attachments 22

      per FEE64 Rate spectra - attachment 23

      Merger etc - attachments 24

      ucesb - attachment 25
       spill length 3.5s, extraction 1.5s

09.50 per FEE64 1.8.W spectra - 20us FSR - attachments 26-27
       no significant change in electronic noise cf. start of run

      per p+n FEE64 1.8.L spectra - attachment 28
       aida09 pulser peak width 62 ch FWHM
       common x/y scale - peak heoght proxy for width


13.57 DAQ continues file R21_141

      DSSSD bias & leakage current - Grafana - attachments 29-30

      FEE64 temperatures OK - attachment 31

      ADC data item stats OK - attachments 32

      per p+n FEE64 1.8.L spectra - attachment 33
       aida09 pulser peak width 55 ch FWHM (~39keV FWHM)

      per FEE64 Rate spectra - attachment 34

      Merger etc - attachments 35

      ucesb - attachment 36
       spill length 3.5s, extraction 1.5s

14.15 analysis data file R21_144 - attachment 35
      max deadtime 16% (aida04)
      no timewarps
      HEC data item rate 2.0kHz
Attachment 1: Screenshot_from_2024-04-27_06-00-39.png
Screenshot_from_2024-04-27_06-00-39.png
Attachment 2: Screenshot_from_2024-04-27_06-01-02.png
Screenshot_from_2024-04-27_06-01-02.png
Attachment 3: Screenshot_from_2024-04-27_06-01-38.png
Screenshot_from_2024-04-27_06-01-38.png
Attachment 4: Screenshot_from_2024-04-27_06-02-22.png
Screenshot_from_2024-04-27_06-02-22.png
Attachment 5: Screenshot_from_2024-04-27_06-02-46.png
Screenshot_from_2024-04-27_06-02-46.png
Attachment 6: Screenshot_from_2024-04-27_06-03-00.png
Screenshot_from_2024-04-27_06-03-00.png
Attachment 7: Screenshot_from_2024-04-27_06-03-32.png
Screenshot_from_2024-04-27_06-03-32.png
Attachment 8: Screenshot_from_2024-04-27_06-04-34.png
Screenshot_from_2024-04-27_06-04-34.png
Attachment 9: Screenshot_from_2024-04-27_06-05-03.png
Screenshot_from_2024-04-27_06-05-03.png
Attachment 10: Screenshot_from_2024-04-27_06-05-37.png
Screenshot_from_2024-04-27_06-05-37.png
Attachment 11: Screenshot_from_2024-04-27_06-06-17.png
Screenshot_from_2024-04-27_06-06-17.png
Attachment 12: Screenshot_from_2024-04-27_06-07-30.png
Screenshot_from_2024-04-27_06-07-30.png
Attachment 13: Screenshot_from_2024-04-27_06-08-12.png
Screenshot_from_2024-04-27_06-08-12.png
Attachment 14: Screenshot_from_2024-04-27_06-11-26.png
Screenshot_from_2024-04-27_06-11-26.png
Attachment 15: Screenshot_from_2024-04-27_06-12-32.png
Screenshot_from_2024-04-27_06-12-32.png
Attachment 16: Screenshot_from_2024-04-27_06-13-07.png
Screenshot_from_2024-04-27_06-13-07.png
Attachment 17: Screenshot_from_2024-04-27_06-13-53.png
Screenshot_from_2024-04-27_06-13-53.png
Attachment 18: Screenshot_from_2024-04-27_06-14-37.png
Screenshot_from_2024-04-27_06-14-37.png
Attachment 19: R20_748
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257504742 ( 1406851.0 Hz)
        Other data format:    4415258 (   24122.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1610 (       8.8 Hz)
                               RESUME:       1611 (       8.8 Hz)
                              SYNC100:      32682 (     178.6 Hz)
                              WR48-63:      32682 (     178.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4346673 (   23747.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     386233 (    2110.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      183.036 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.284                0.000
  1               10.840                0.000
  2                0.188                0.000
  3               23.217                0.000
  4                0.100                0.000
  5                2.070                0.000
  6                0.000                0.000
  7                2.151                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                8.754                0.000
 11                0.009                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.015                0.000
 15                2.919                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   12463516       8663          0          0         35         35       1556       1556          0       5481          0      27718
  1   18658776     198614          0          0        164        164       2390       2390          0     193506          0      51705
  2   15562218    1904264          0          0         13         13       2088       2088          0    1900062          0      19441
  3   28663274    1575353          0          0        203        204       3714       3714          0    1567518          0      14524
  4    7649540     297936          0          0          9          9        969        969          0     295980          0      20940
  5   22941092     390586          0          0        220        220       3010       3010          0     384126          0      18537
  6    5869094       1360          0          0          0          0        680        680          0          0          0       5932
  7   40428040      10412          0          0        122        122       5084       5084          0          0          0     148981
  8    2103382        490          0          0          0          0        245        245          0          0          0      10601
  9    5530471       1492          0          0          0          0        746        746          0          0          0       4423
 10   31486867       9242          0          0        711        711       3910       3910          0          0          0       1813
 11    6680681       1722          0          0          2          2        859        859          0          0          0      37346
 12    4842246       1236          0          0          0          0        618        618          0          0          0       5812
 13    3504495        918          0          0          0          0        459        459          0          0          0       6910
 14    6454647       1674          0          0          3          3        834        834          0          0          0       4246
 15   44666403      11296          0          0        128        128       5520       5520          0          0          0       7304
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   38.639s (  828.186 blocks/s,  51.762 Mb/s)
Attachment 20: Capture.PNG
Capture.PNG
Attachment 21: Screenshot_from_2024-04-27_09-32-29.png
Screenshot_from_2024-04-27_09-32-29.png
Attachment 22: Screenshot_from_2024-04-27_09-33-31.png
Screenshot_from_2024-04-27_09-33-31.png
Attachment 23: Screenshot_from_2024-04-27_09-34-20.png
Screenshot_from_2024-04-27_09-34-20.png
Attachment 24: Screenshot_from_2024-04-27_09-34-58.png
Screenshot_from_2024-04-27_09-34-58.png
Attachment 25: Screenshot_from_2024-04-27_09-35-41.png
Screenshot_from_2024-04-27_09-35-41.png
Attachment 26: Screenshot_from_2024-04-27_09-47-09.png
Screenshot_from_2024-04-27_09-47-09.png
Attachment 27: Screenshot_from_2024-04-27_09-48-06.png
Screenshot_from_2024-04-27_09-48-06.png
Attachment 28: Screenshot_from_2024-04-27_09-52-52.png
Screenshot_from_2024-04-27_09-52-52.png
Attachment 29: Screenshot_from_2024-04-27_13-57-44.png
Screenshot_from_2024-04-27_13-57-44.png
Attachment 30: Screenshot_from_2024-04-27_13-58-01.png
Screenshot_from_2024-04-27_13-58-01.png
Attachment 31: Screenshot_from_2024-04-27_14-01-26.png
Screenshot_from_2024-04-27_14-01-26.png
Attachment 32: Screenshot_from_2024-04-27_13-58-55.png
Screenshot_from_2024-04-27_13-58-55.png
Attachment 33: Screenshot_from_2024-04-27_14-03-49.png
Screenshot_from_2024-04-27_14-03-49.png
Attachment 34: Screenshot_from_2024-04-27_14-04-30.png
Screenshot_from_2024-04-27_14-04-30.png
Attachment 35: Screenshot_from_2024-04-27_14-05-30.png
Screenshot_from_2024-04-27_14-05-30.png
Attachment 36: Screenshot_from_2024-04-27_14-06-42.png
Screenshot_from_2024-04-27_14-06-42.png
Attachment 37: R21_144
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256251858 ( 1109842.9 Hz)
        Other data format:    5668143 (   24549.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1144 (       5.0 Hz)
                               RESUME:       1144 (       5.0 Hz)
                              SYNC100:      32860 (     142.3 Hz)
                              WR48-63:      32860 (     142.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5600135 (   24254.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     452307 (    1959.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      230.890 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.168                0.000
  1               16.872                0.000
  2                0.114                0.000
  3               36.051                0.000
  4                0.888                0.000
  5                2.369                0.000
  6                0.000                0.000
  7                1.443                0.000
  8                0.006                0.000
  9                0.000                0.000
 10                1.304                0.000
 11                0.023                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.033                0.000
 15                1.208                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7949000       9111          0          0         19         19       1076       1076          0       6921          0      39332
  1   26808065     269780          0          0        235        235       3356       3356          0     262598          0      70729
  2   11614466    2448126          0          0         13         13       1817       1817          0    2444466          0      27115
  3   37929923    1952070          0          0        294        294       4896       4896          0    1941690          0      19677
  4    8915163     414541          0          0         12         12       1163       1163          0     412191          0      27553
  5   22729514     538305          0          0        114        114       2904       2904          0     532269          0      25872
  6    5822184       1488          0          0          0          0        744        744          0          0          0       8358
  7   40443466      10506          0          0         99         99       5154       5154          0          0          0     123022
  8    3043811        744          0          0          1          1        371        371          0          0          0      14593
  9    5157550       1312          0          0          0          0        656        656          0          0          0       6148
 10   26044004       6988          0          0        273        273       3221       3221          0          0          0       2629
 11    5888316       1514          0          0          4          4        753        753          0          0          0      52905
 12    5831030       1426          0          0          0          0        713        713          0          0          0       8360
 13    4114996        986          0          0          0          0        493        493          0          0          0       9929
 14    5361222       1380          0          0          5          5        685        685          0          0          0       5844
 15   38599148       9866          0          0         75         75       4858       4858          0          0          0      10241
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.879s (  944.541 blocks/s,  59.034 Mb/s)
  611   Fri Apr 26 23:04:51 2024 Marc00:00-08:00 Saturday 27 April

It's a new day, it's a new shift and I'm feeeeeliiiiing goooood !!!!!

0.15 Checks:
      DSSSD bias & leakage current - Grafana - attachments 1-2
       leakage current dropped a bit since the evening shift ~22:00.

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

      ucesb - attachment 7

2:20 Nothing new to report. All good. Leakage currents continue to go down slowly. See attachement 8 to 12

3:30 Still running smoothly. 

5:20 AIDA 4 stop counting in the last 20 min. I informed the main DAQ and they started a new run without AIDA.

        DAQ restarted OK. Main DAQ informed.

 

Attachment 1: Screenshot_from_2024-04-27_00-17-00.png
Screenshot_from_2024-04-27_00-17-00.png
Attachment 2: Screenshot_from_2024-04-27_00-16-16.png
Screenshot_from_2024-04-27_00-16-16.png
Attachment 3: Screenshot_from_2024-04-27_00-18-49.png
Screenshot_from_2024-04-27_00-18-49.png
Attachment 4: Screenshot_from_2024-04-27_00-19-05.png
Screenshot_from_2024-04-27_00-19-05.png
Attachment 5: Screenshot_from_2024-04-27_00-19-19.png
Screenshot_from_2024-04-27_00-19-19.png
Attachment 6: Screenshot_from_2024-04-27_00-21-44.png
Screenshot_from_2024-04-27_00-21-44.png
Attachment 7: Screenshot_from_2024-04-27_00-21-00.png
Screenshot_from_2024-04-27_00-21-00.png
Attachment 8: Screenshot_from_2024-04-27_02-32-22.png
Screenshot_from_2024-04-27_02-32-22.png
Attachment 9: Screenshot_from_2024-04-27_02-37-20.png
Screenshot_from_2024-04-27_02-37-20.png
Attachment 10: Screenshot_from_2024-04-27_02-37-31.png
Screenshot_from_2024-04-27_02-37-31.png
Attachment 11: Screenshot_from_2024-04-27_02-40-39.png
Screenshot_from_2024-04-27_02-40-39.png
Attachment 12: Screenshot_from_2024-04-27_02-41-13.png
Screenshot_from_2024-04-27_02-41-13.png
  610   Fri Apr 26 15:34:00 2024 RDP16:00 - 00:00 Friday 26 April

This appeared in the system log:

Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

These messages seem to appear sporadically.

18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?

18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.

aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters

Thanks Nic!

22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)

22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.

 

Attachment 1: Screenshot_2024-04-26_at_16-40-03_ucesb.png
Screenshot_2024-04-26_at_16-40-03_ucesb.png
Attachment 2: Screenshot_2024-04-26_at_16-41-57_AIDA_-_Grafana.png
Screenshot_2024-04-26_at_16-41-57_AIDA_-_Grafana.png
Attachment 3: Screenshot_2024-04-26_at_19-07-21_Merger_Input_Data_Link_Rates_aidas-gsi.png
Screenshot_2024-04-26_at_19-07-21_Merger_Input_Data_Link_Rates_aidas-gsi.png
Attachment 4: Screenshot_2024-04-26_at_19-18-28_ucesb.png
Screenshot_2024-04-26_at_19-18-28_ucesb.png
Attachment 5: Screenshot_2024-04-26_at_22-12-06_ucesb.png
Screenshot_2024-04-26_at_22-12-06_ucesb.png
Attachment 6: Screenshot_2024-04-26_at_22-11-47_AIDA_-_Grafana.png
Screenshot_2024-04-26_at_22-11-47_AIDA_-_Grafana.png
  609   Fri Apr 26 08:15:59 2024 TD[How To] Recover DAQ when one FEE64 stops sending data
Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running *and* data is being transferred to the Merger and TapeServer.

Typical symptoms are:

- no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser']
- zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics']
- Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control']
   active data links are green, inactive links are red
   N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n

0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5) Reboot problem FEE64s
   Launch a command terminal and telnet to the problem FEE64 for example

    telnet aida04

   Enter user 'root' and enter password. When logged in reboot the FEE64 using the command

    reboot

6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal']
    you should see two messages showing the FEE64 mounting filesystems
    wait c. three minutes for boot to complete

7) Restart DAQ

   Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control']
   Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control']
   Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control']
   Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control']
   Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control']
   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

8) Restore ASIC settings

   Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control']
   Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings']
   Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings']
   Wait 10s

9) Check ASIC settings

   Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control']
   Wait for popup indicating completion of checks and dismiss

10) System wide checks

    Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']

    Select 'Check clock status'  [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks']
     make a note or take a screenshot of any error messages observed

11) Start DAQ

    Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control']
    Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

13) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

14) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

15) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.








    








   

   




   
  608   Fri Apr 26 07:01:26 2024 Norah08:00 - 16:00 Friday 26 April

10:00 Checks:

All seems smooth.

DSSD bias and  leakage current ( Attachment 1)

FEE64 temperatures (Attachment 2)

Per FEE64 Rate spectra (Attachment 3)

ADC data item stats -(Attachment 4)

Ucesb (Attachment 5)

 

10:21

Aida05 was zero not reading had to call Nic to fix it.

10:34

Now, Aida05 is running well.

.......................................................

12:00 Checks:

Everything appears to be going smoothly, and I took screenshot of them.

 

12:28, Aida02 showed as not reading , had to email Nic to fix it.

12:30, it resumed reading and returned to normal operation.

.........................................................

14:00 Checks:

Everything is going well , and I have captured screenshots of each one.

......................................................

16:00 Checks:

All seems smooth, and I took screenshot of them.

 

 

 

 

 

Attachment 1: Screenshot_from_2024-04-26_10-00-39.png
Screenshot_from_2024-04-26_10-00-39.png
Attachment 2: Screenshot_from_2024-04-26_10-02-35.png
Screenshot_from_2024-04-26_10-02-35.png
Attachment 3: Screenshot_from_2024-04-26_10-04-40.png
Screenshot_from_2024-04-26_10-04-40.png
Attachment 4: Screenshot_from_2024-04-26_10-06-18.png
Screenshot_from_2024-04-26_10-06-18.png
Attachment 5: Screenshot_from_2024-04-26_10-09-52.png
Screenshot_from_2024-04-26_10-09-52.png
  607   Thu Apr 25 22:45:30 2024 Tom, Muneerah, NorahOver night shift AIDA 26 Fri

 

01:02

Aida04 was zero not reading had to call Tom to fix it.

02:00 checked all good and attached.

Its been running good after that till 7:30 again the same problem happened and Tom fixed it.

 

 

Attachment 1: Screenshot_from_2024-04-25_23-42-15.png
Screenshot_from_2024-04-25_23-42-15.png
Attachment 2: Screenshot_from_2024-04-26_00-05-47.png
Screenshot_from_2024-04-26_00-05-47.png
Attachment 3: Screenshot_from_2024-04-26_00-05-15.png
Screenshot_from_2024-04-26_00-05-15.png
Attachment 4: Screenshot_from_2024-04-26_00-04-22.png
Screenshot_from_2024-04-26_00-04-22.png
Attachment 5: Screenshot_from_2024-04-26_02-00-17.png
Screenshot_from_2024-04-26_02-00-17.png
Attachment 6: Screenshot_from_2024-04-26_01-58-39.png
Screenshot_from_2024-04-26_01-58-39.png
Attachment 7: Screenshot_from_2024-04-26_01-58-39.png
Screenshot_from_2024-04-26_01-58-39.png
Attachment 8: Screenshot_from_2024-04-26_01-57-38.png
Screenshot_from_2024-04-26_01-57-38.png
Attachment 9: Screenshot_from_2024-04-26_01-57-01.png
Screenshot_from_2024-04-26_01-57-01.png
  606   Thu Apr 25 15:02:50 2024 Marc16:00 - 0:00 Thursday 25 April
16:20: 
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
             

16.45 Checks:
      DSSSD bias & leakage current - Grafana - attachments 2
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 3

      ADC data item stats - attachment 5

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 6

      ucesb - attachment 7

 

XX.XX Checks

       DSSSD bias & leakage current - Grafana - attachments X

       FEE64 temperatures OK - attachment X

       ADC data item stats - attachment X

       per FEE64 Rate spectra - attachment X

       Merger etc - attachment X

       ucesb - attachment X

Attachment 1: Screenshot_from_2024-04-25_16-19-32.png
Screenshot_from_2024-04-25_16-19-32.png
Attachment 2: Screenshot_from_2024-04-25_16-45-19.png
Screenshot_from_2024-04-25_16-45-19.png
Attachment 3: Screenshot_from_2024-04-25_16-46-12.png
Screenshot_from_2024-04-25_16-46-12.png
Attachment 4: Screenshot_from_2024-04-25_16-46-41.png
Screenshot_from_2024-04-25_16-46-41.png
Attachment 5: Screenshot_from_2024-04-25_16-47-00.png
Screenshot_from_2024-04-25_16-47-00.png
Attachment 6: Screenshot_from_2024-04-25_16-52-18.png
Screenshot_from_2024-04-25_16-52-18.png
Attachment 7: Screenshot_from_2024-04-25_16-52-50.png
Screenshot_from_2024-04-25_16-52-50.png
Attachment 8: Screenshot_from_2024-04-25_17-23-36.png
Screenshot_from_2024-04-25_17-23-36.png
  605   Thu Apr 25 08:26:50 2024 TD08:00-16:00 Thursday 25 April
09.56 DAQ continues data file S100_alpha/R18_387
      FRS switching settings

      DSSSD bias & leakage current OK - attachments 1

      FEE64 temperatures OK  - attachment 2
       aida02 ASIC temp - known fault

      All system wide checks OK *except* as shown - attachments 3-5

      WR timestamps OK - attachment 6

      ADC, PAUSE, RESUME & Correlation Scaler data time stats - attachments 7-10

      per FEE64 Rate spectra - attachments 11

      per p+n FEE64 1.8.L spectra - attachments 12-13
       aida09 pulser peak width 57 ch FWHM 
       common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths

      per n+n FEE64 1.8.L spectra - attachment 14

      per FEE64 1.8.H spectra - attachments 15-16

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-18
       only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?

      Merger, TapeServer etc - attachments 19-21

      BNC PB-5 settings - attachment 22

10.07 data file S100_alpha/R20
      new run for new FRS setting

10.08 all histograms, stats, merger stats and data link stats zero'd

10.10 ADC data item stats (beam OFF) - attachment 23
      11/16 < 20k, max 129k (aida08)

10.31 analysis of data file R18_349 (closed 06.00 this morning)
      max deadtime 12% (aida04)
      no timewarps
      HEC data item rate 1.7kHz

11.00 tuning 168dy FRS setting
      ucesb - attachment 24
      spill length 3.5s, extraction time 1.5s

11.30 degraders now set to implant in AIDA DSSSDs
      high rates expected 

      per FEE64 Rate spectra - attachment 25

      ADC data item stats - attachment 26

      Merger etc - attachment 27
      in spill rates to MBS up 30+Mb/s, 2.5M data item/s merged

      DSSSD bias & leakage current - attachment 28
       c. few hundred nA step in leakage current 

      analysis of data file R20_33 - attachment 29
      max deadtime 18% (aida08)
      no timewarps
      HEC data item rate 4.1kHz (c. 2.4x previous FRS setting)

      Correcting for spill cycle => 5.9kHz in spill (online analysis shows no HEC events observed outside spill)
      assume typical HEC event multiplicity = 3 => implant rate ~ 2.0kHz in spill or 6800 implants per spill
      SC42 L/R currently report c. 4.5kHz

      Note that we currently see a higher fraction of events in DSSSD#2 cf. previous setting so multiplicity assumption may be low
      If we assume multipicity = 6 => implant rate ~ 1.0kHz in spill or 3200 implants per spill

      we are probably somewhere in between

12.50 DSSSD bias & leakage current - Grafana - attachment 30
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 31

      ADC data item stats - attachment 32

      per FEE64 Rate spectra - attachment 33

      Merger etc - attachment 34

14.56 analysis of data file R20_106 - attachment 35
      max deadtime c. 18% (aida04 & aida08), c. 12% (aida03), c. 11% (aida06), c. 9% (aida02 & aida13), c. 6% (aida05), all others <2%
      no timewarps
      HEC data item rate 4.2kHz


15.40 DSSSD bias & leakage current - Grafana - attachments 36-37
       leakage current ramping, spill micro structure observable

      FEE64 temperatures OK - attachment 38

      ADC data item stats - attachment 39

      per FEE64 Rate spectra - attachment 40

      Merger etc - attachment 41

      ucesb - attachment 42

15.57 all histograms zero'd




      
Attachment 1: Screenshot_from_2024-04-25_09-27-20.png
Screenshot_from_2024-04-25_09-27-20.png
Attachment 2: Screenshot_from_2024-04-25_09-27-44.png
Screenshot_from_2024-04-25_09-27-44.png
Attachment 3: Screenshot_from_2024-04-25_09-28-05.png
Screenshot_from_2024-04-25_09-28-05.png
Attachment 4: Screenshot_from_2024-04-25_09-28-16.png
Screenshot_from_2024-04-25_09-28-16.png
Attachment 5: Screenshot_from_2024-04-25_09-28-23.png
Screenshot_from_2024-04-25_09-28-23.png
Attachment 6: Screenshot_from_2024-04-25_09-29-04.png
Screenshot_from_2024-04-25_09-29-04.png
Attachment 7: Screenshot_from_2024-04-25_09-29-20.png
Screenshot_from_2024-04-25_09-29-20.png
Attachment 8: Screenshot_from_2024-04-25_09-29-46.png
Screenshot_from_2024-04-25_09-29-46.png
Attachment 9: Screenshot_from_2024-04-25_09-29-59.png
Screenshot_from_2024-04-25_09-29-59.png
Attachment 10: Screenshot_from_2024-04-25_09-30-15.png
Screenshot_from_2024-04-25_09-30-15.png
Attachment 11: Screenshot_from_2024-04-25_09-30-56.png
Screenshot_from_2024-04-25_09-30-56.png
Attachment 12: Screenshot_from_2024-04-25_09-31-55.png
Screenshot_from_2024-04-25_09-31-55.png
Attachment 13: Screenshot_from_2024-04-25_09-32-11.png
Screenshot_from_2024-04-25_09-32-11.png
Attachment 14: Screenshot_from_2024-04-25_09-33-15.png
Screenshot_from_2024-04-25_09-33-15.png
Attachment 15: Screenshot_from_2024-04-25_09-34-23.png
Screenshot_from_2024-04-25_09-34-23.png
Attachment 16: Screenshot_from_2024-04-25_09-34-52.png
Screenshot_from_2024-04-25_09-34-52.png
Attachment 17: Screenshot_from_2024-04-25_09-35-38.png
Screenshot_from_2024-04-25_09-35-38.png
Attachment 18: Screenshot_from_2024-04-25_09-36-14.png
Screenshot_from_2024-04-25_09-36-14.png
Attachment 19: Screenshot_from_2024-04-25_09-36-28.png
Screenshot_from_2024-04-25_09-36-28.png
Attachment 20: Screenshot_from_2024-04-25_09-37-02.png
Screenshot_from_2024-04-25_09-37-02.png
Attachment 21: Screenshot_from_2024-04-25_09-37-17.png
Screenshot_from_2024-04-25_09-37-17.png
Attachment 22: Screenshot_from_2024-04-25_10-02-31.png
Screenshot_from_2024-04-25_10-02-31.png
Attachment 23: R18_349
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254055879 (  798496.4 Hz)
        Other data format:    7864121 (   24716.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        850 (       2.7 Hz)
                               RESUME:        851 (       2.7 Hz)
                              SYNC100:      33185 (     104.3 Hz)
                              WR48-63:      33185 (     104.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7796050 (   24502.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     547807 (    1721.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      318.168 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.011                0.000
  1               15.547                0.000
  2                0.026                0.000
  3               39.441                0.000
  4                0.410                0.000
  5                0.954                0.000
  6                0.000                0.000
  7                0.601                0.000
  8                0.003                0.000
  9                0.000                0.000
 10                0.050                0.000
 11                0.014                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.017                0.000
 15                0.116                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6649451      11230          0          0          3          3        839        839          0       9546          0      49433
  1   33550123     378474          0          0        245        245       4373       4373          0     369238          0     100748
  2    7942952    3356106          0          0          5          5       1429       1429          0    3353238          0      35304
  3   50066846    2782463          0          0        379        380       6547       6547          0    2768610          0      28219
  4   11634489     554162          0          0         19         19       1542       1542          0     551040          0      47658
  5   18478696     749338          0          0         77         77       2403       2403          0     744378          0      37331
  6    6034652       1500          0          0          0          0        750        750          0          0          0      11345
  7   41275133      10854          0          0         71         71       5356       5356          0          0          0      92779
  8    3808950        934          0          0          2          2        465        465          0          0          0      19548
  9    5816078       1448          0          0          0          0        724        724          0          0          0       8443
 10   16595017       4342          0          0         16         16       2155       2155          0          0          0       3675
 11    5732285       1360          0          0          2          2        678        678          0          0          0      65221
 12    7170732       1770          0          0          0          0        885        885          0          0          0      12515
 13    5117877       1350          0          0          0          0        675        675          0          0          0      13502
 14    5096608       1234          0          0          2          2        615        615          0          0          0       7339
 15   29085990       7556          0          0         29         29       3749       3749          0          0          0      14747
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.840s ( 1005.030 blocks/s,  62.814 Mb/s)
Attachment 24: Screenshot_from_2024-04-25_11-27-43.png
Screenshot_from_2024-04-25_11-27-43.png
Attachment 25: Screenshot_from_2024-04-25_11-39-59.png
Screenshot_from_2024-04-25_11-39-59.png
Attachment 26: Screenshot_from_2024-04-25_11-40-11.png
Screenshot_from_2024-04-25_11-40-11.png
Attachment 27: Screenshot_from_2024-04-25_11-40-28.png
Screenshot_from_2024-04-25_11-40-28.png
Attachment 28: Screenshot_from_2024-04-25_11-43-22.png
Screenshot_from_2024-04-25_11-43-22.png
Attachment 29: R20_33
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257467881 ( 1603138.0 Hz)
        Other data format:    4452119 (   27721.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1792 (      11.2 Hz)
                               RESUME:       1788 (      11.1 Hz)
                              SYNC100:      32598 (     203.0 Hz)
                              WR48-63:      32598 (     203.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4383343 (   27293.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     659812 (    4108.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.602 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                4.032                0.000
  1               14.271                0.000
  2               17.643                0.000
  3               26.583                0.000
  4               10.476                0.000
  5               15.929                0.000
  6                1.815                0.000
  7               29.387                0.000
  8                0.158                0.000
  9                0.036                0.000
 10                0.592                0.000
 11                0.531                0.000
 12               14.310                0.000
 13                0.165                0.000
 14                0.189                0.000
 15                0.995                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14126134       8441          0          0         70         70       1791       1791          0       4719          0      78615
  1   16495785     288035          0          0        163        162       2094       2094          0     283522          0     105734
  2   19988207    1651337          0          0        161        161       2732       2732          0    1645551          0      49977
  3   24594867    1476087          0          0        215        214       3183       3183          0    1469292          0      26591
  4   15580694     468076          0          0        119        119       1941       1941          0     463956          0      27262
  5   22750595     522722          0          0        300        299       2910       2910          0     516303          0      62407
  6   11629769       3045          0          0         37         36       1486       1486          0          0          0      11991
  7   37179788       9710          0          0        273        273       4582       4582          0          0          0      86342
  8    7628214       1994          0          0          9          9        988        988          0          0          0       1179
  9    5725517       1418          0          0          5          5        704        704          0          0          0       1622
 10   13190745       3550          0          0         69         69       1706       1706          0          0          0         70
 11    9285908       2326          0          0         29         29       1134       1134          0          0          0      74999
 12   16713447       4700          0          0        252        252       2098       2098          0          0          0      21646
 13    6115493       1498          0          0         25         25        724        724          0          0          0      35370
 14    8494444       2072          0          0         11         11       1025       1025          0          0          0         92
 15   27968274       7108          0          0         54         54       3500       3500          0          0          0      75915
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.629s (  924.083 blocks/s,  57.755 Mb/s)
Attachment 30: Screenshot_from_2024-04-25_12-50-09.png
Screenshot_from_2024-04-25_12-50-09.png
Attachment 31: Screenshot_from_2024-04-25_12-50-16.png
Screenshot_from_2024-04-25_12-50-16.png
Attachment 32: Screenshot_from_2024-04-25_12-50-28.png
Screenshot_from_2024-04-25_12-50-28.png
Attachment 33: Screenshot_from_2024-04-25_12-50-35.png
Screenshot_from_2024-04-25_12-50-35.png
Attachment 34: Screenshot_from_2024-04-25_12-51-02.png
Screenshot_from_2024-04-25_12-51-02.png
Attachment 35: R20_106
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257522900 ( 1655989.1 Hz)
        Other data format:    4397100 (   28275.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1842 (      11.8 Hz)
                               RESUME:       1844 (      11.9 Hz)
                              SYNC100:      32579 (     209.5 Hz)
                              WR48-63:      32579 (     209.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4328256 (   27832.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     659185 (    4238.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      155.510 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                3.085                0.000
  1               13.568                0.000
  2               18.124                0.000
  3               27.697                0.000
  4                9.792                0.000
  5               16.474                0.000
  6                2.671                0.000
  7               28.543                0.000
  8                0.065                0.000
  9                0.015                0.000
 10                0.643                0.000
 11                0.597                0.000
 12               14.389                0.000
 13                0.083                0.000
 14                0.439                0.000
 15                1.481                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13689199       8238          0          0         65         65       1771       1771          0       4566          0      77953
  1   16947527     287988          0          0        165        166       2170       2170          0     283317          0     104868
  2   20356284    1630731          0          0        174        175       2797       2797          0    1624788          0      51128
  3   23896908    1435934          0          0        213        214       3077       3077          0    1429353          0      26122
  4   15647459     464207          0          0        117        117       2008       2008          0     459957          0      27024
  5   22696175     532714          0          0        306        305       2914       2914          0     526275          0      64287
  6   12236208       3082          0          0         42         42       1499       1499          0          0          0      11994
  7   36272075       9624          0          0        259        259       4553       4553          0          0          0      84854
  8    7567267       1906          0          0          4          4        949        949          0          0          0       1286
  9    5993294       1442          0          0          3          3        718        718          0          0          0       1730
 10   13223300       3544          0          0         82         82       1690       1690          0          0          0         53
 11    9630981       2436          0          0         36         36       1182       1182          0          0          0      75223
 12   16792007       4646          0          0        270        270       2053       2053          0          0          0      21202
 13    6155506       1514          0          0         19         19        738        738          0          0          0      35333
 14    8544527       2006          0          0         19         19        984        984          0          0          0         68
 15   27874183       7088          0          0         68         68       3476       3476          0          0          0      76060
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.426s ( 1051.740 blocks/s,  65.734 Mb/s)
Attachment 36: Screenshot_from_2024-04-25_15-39-07.png
Screenshot_from_2024-04-25_15-39-07.png
Attachment 37: Screenshot_from_2024-04-25_15-39-14.png
Screenshot_from_2024-04-25_15-39-14.png
Attachment 38: Screenshot_from_2024-04-25_15-39-36.png
Screenshot_from_2024-04-25_15-39-36.png
Attachment 39: Screenshot_from_2024-04-25_15-39-53.png
Screenshot_from_2024-04-25_15-39-53.png
Attachment 40: Screenshot_from_2024-04-25_15-40-30.png
Screenshot_from_2024-04-25_15-40-30.png
Attachment 41: Screenshot_from_2024-04-25_15-40-48.png
Screenshot_from_2024-04-25_15-40-48.png
Attachment 42: Screenshot_from_2024-04-25_15-43-08.png
Screenshot_from_2024-04-25_15-43-08.png
  604   Wed Apr 24 21:39:51 2024 TD00:00-08:00 Thursday 25 April
22.43 24.4.24

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 rate spectra - attachments 4

      Merger etc - attachment 5

      analysis of data file R18_266 - attachment 6
       max dead time 12% (aida04)
       no timewarps
       HEC data item rate 1.7kHz

03.37 DSSSD bias & leakage current OK - attachments 7-8

      FEE64 temperatures OK - attachment 9

      ADC data item stats - attachment 10

      per FEE64 rate spectra - attachments 11

      Merger etc - attachment 12
Attachment 1: Screenshot_from_2024-04-24_22-43-23.png
Screenshot_from_2024-04-24_22-43-23.png
Attachment 2: Screenshot_from_2024-04-24_22-43-58.png
Screenshot_from_2024-04-24_22-43-58.png
Attachment 3: Screenshot_from_2024-04-24_22-44-50.png
Screenshot_from_2024-04-24_22-44-50.png
Attachment 4: Screenshot_from_2024-04-24_22-45-35.png
Screenshot_from_2024-04-24_22-45-35.png
Attachment 5: Screenshot_from_2024-04-24_22-46-21.png
Screenshot_from_2024-04-24_22-46-21.png
Attachment 6: R18_266
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254416606 (  820824.6 Hz)
        Other data format:    7503396 (   24208.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        846 (       2.7 Hz)
                               RESUME:        846 (       2.7 Hz)
                              SYNC100:      33155 (     107.0 Hz)
                              WR48-63:      33155 (     107.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7435394 (   23988.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     524404 (    1691.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.952 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.014                0.000
  1               16.009                0.000
  2                0.003                0.000
  3               37.388                0.000
  4                0.176                0.000
  5                0.544                0.000
  6                0.000                0.000
  7                0.784                0.000
  8                0.015                0.000
  9                0.000                0.000
 10                0.083                0.000
 11                0.004                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.017                0.000
 15                0.227                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6529814      10932          0          0          4          4        812        812          0       9300          0      46302
  1   32238460     350302          0          0        231        231       4129       4129          0     341582          0      92776
  2    8209450    3208628          0          0          2          2       1421       1421          0    3205782          0      32990
  3   48353626    2676061          0          0        374        374       6369       6369          0    2662575          0      27015
  4   10911439     520528          0          0          9          9       1454       1454          0     517602          0      45169
  5   18550590     703657          0          0         64         64       2488       2488          0     698553          0      35506
  6    6051492       1578          0          0          0          0        789        789          0          0          0      10690
  7   41812278      11146          0          0         85         85       5488       5488          0          0          0      97961
  8    3594476        900          0          0          4          4        446        446          0          0          0      18281
  9    5838926       1490          0          0          0          0        745        745          0          0          0       8125
 10   18621964       4722          0          0         27         27       2334       2334          0          0          0       3637
 11    5447469       1386          0          0          1          1        692        692          0          0          0      60590
 12    7038926       1642          0          0          0          0        821        821          0          0          0      12101
 13    5011795       1240          0          0          0          0        620        620          0          0          0      12487
 14    5028304       1286          0          0          2          2        641        641          0          0          0       7019
 15   31177597       7898          0          0         43         43       3906       3906          0          0          0      13755
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.508s (  984.379 blocks/s,  61.524 Mb/s)
Attachment 7: Screenshot_from_2024-04-25_03-37-40.png
Screenshot_from_2024-04-25_03-37-40.png
Attachment 8: Screenshot_from_2024-04-25_03-38-57.png
Screenshot_from_2024-04-25_03-38-57.png
Attachment 9: Screenshot_from_2024-04-25_03-39-38.png
Screenshot_from_2024-04-25_03-39-38.png
Attachment 10: Screenshot_from_2024-04-25_03-40-20.png
Screenshot_from_2024-04-25_03-40-20.png
Attachment 11: Screenshot_from_2024-04-25_03-40-52.png
Screenshot_from_2024-04-25_03-40-52.png
Attachment 12: Screenshot_from_2024-04-25_03-41-31.png
Screenshot_from_2024-04-25_03-41-31.png
  603   Wed Apr 24 18:44:05 2024 PP19:30 checks

All seems smooth.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-24_19-34-58.png
Screenshot_from_2024-04-24_19-34-58.png
Attachment 2: Screenshot_from_2024-04-24_19-37-12.png
Screenshot_from_2024-04-24_19-37-12.png
Attachment 3: Screenshot_from_2024-04-24_19-40-36.png
Screenshot_from_2024-04-24_19-40-36.png
Attachment 4: Screenshot_from_2024-04-24_19-41-39.png
Screenshot_from_2024-04-24_19-41-39.png
Attachment 5: Screenshot_from_2024-04-24_19-42-20.png
Screenshot_from_2024-04-24_19-42-20.png
Attachment 6: Screenshot_from_2024-04-24_19-42-53.png
Screenshot_from_2024-04-24_19-42-53.png
Attachment 7: Screenshot_from_2024-04-24_19-43-21.png
Screenshot_from_2024-04-24_19-43-21.png
Attachment 8: Screenshot_from_2024-04-24_19-43-41.png
Screenshot_from_2024-04-24_19-43-41.png
  602   Wed Apr 24 15:37:59 2024 PP16:00-00:00 shift Wednesday 24 April

All seems OK.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-24_16-38-45.png
Screenshot_from_2024-04-24_16-38-45.png
Attachment 2: Screenshot_from_2024-04-24_16-39-09.png
Screenshot_from_2024-04-24_16-39-09.png
Attachment 3: Screenshot_from_2024-04-24_16-39-38.png
Screenshot_from_2024-04-24_16-39-38.png
Attachment 4: Screenshot_from_2024-04-24_16-40-24.png
Screenshot_from_2024-04-24_16-40-24.png
Attachment 5: Screenshot_from_2024-04-24_16-41-15.png
Screenshot_from_2024-04-24_16-41-15.png
Attachment 6: Screenshot_from_2024-04-24_16-43-41.png
Screenshot_from_2024-04-24_16-43-41.png
Attachment 7: Screenshot_from_2024-04-24_16-44-15.png
Screenshot_from_2024-04-24_16-44-15.png
Attachment 8: Screenshot_from_2024-04-24_16-44-32.png
Screenshot_from_2024-04-24_16-44-32.png
  601   Wed Apr 24 09:04:55 2024 TD08:00-16:00 Wednesday 24 April
09.56 DAQ continues data file S100_alpha/R18_122
      FRS setting 162Eu

      DSSSD bias & leakage current OK - attachments 1-2

      FEE64 temperatures OK  - attachment 3
       aida02 ASIC temp - known fault

      All system wide checks OK *except* as shown - attachments 4-6

      WR timestamps OK - attachment 7

      ASIC settings aida01 & aida02 - attachments 8-9

      ADC, DISC, PAUSE, RESUME, WR & Correlation Scaler data time stats - attachments 10-15

      per FEE64 Rate spectra - attachments 16-18
       unusually can observe HEC Rates esp. in DSSSD #1 

      per p+n FEE64 1.8.L spectra - attachments 19-21
       aida09 pulser peak width 61 ch FWHM 
       common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths

      per n+n FEE64 1.8.L spectra - attachment 22

      per FEE64 1.8.H spectra - attachments 23-24

      per FEE64 1.8.W spectra - 20us FSR - attachments 25-26
       only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?

      Merger, TapeServer etc - attachments 27-28

      ucesb - attachments 29-30
       spill cycle reported as 3.5s, extraction 1.5s

10.36 All histograms & stats zero'd

      BNC PB-5 settings - attachment 31

      DSSSD bias & leakage current - Grafana Alerting - attachment 32
       can observe bias & leakage current in greater detail
       DSSSD #1 bias stable, c. 10s nA micro structure observed in leakage current
       DSSSD #2 bias c. 10mV variations, micro structure observed in leakage current but smaller variations cf. DSSSD#1

10.46 analysis of data file R18_131
      max dead time 12% aida04, 6% aida02, all others 0.1% or less
      no timewarps
      HEC data item rates 1.6kHz - correcting for spill cycle => 2.3kHz in spill (online analysis shows no HEC events observed outside spill)
      assume typical HEC event multiplicity = 3 => implant rate ~ 760Hz in spill or 2700 implants per spill
      SC42 L/R currently report c. 2kHz





   

      
Attachment 1: Screenshot_from_2024-04-24_09-56-46.png
Screenshot_from_2024-04-24_09-56-46.png
Attachment 2: Screenshot_from_2024-04-24_09-57-04.png
Screenshot_from_2024-04-24_09-57-04.png
Attachment 3: Screenshot_from_2024-04-24_09-57-42.png
Screenshot_from_2024-04-24_09-57-42.png
Attachment 4: Screenshot_from_2024-04-24_09-57-57.png
Screenshot_from_2024-04-24_09-57-57.png
Attachment 5: Screenshot_from_2024-04-24_09-58-05.png
Screenshot_from_2024-04-24_09-58-05.png
Attachment 6: Screenshot_from_2024-04-24_09-58-19.png
Screenshot_from_2024-04-24_09-58-19.png
Attachment 7: Screenshot_from_2024-04-24_09-58-57.png
Screenshot_from_2024-04-24_09-58-57.png
Attachment 8: Screenshot_from_2024-04-24_09-59-09.png
Screenshot_from_2024-04-24_09-59-09.png
Attachment 9: Screenshot_from_2024-04-24_09-59-20.png
Screenshot_from_2024-04-24_09-59-20.png
Attachment 10: Screenshot_from_2024-04-24_09-59-47.png
Screenshot_from_2024-04-24_09-59-47.png
Attachment 11: Screenshot_from_2024-04-24_09-59-59.png
Screenshot_from_2024-04-24_09-59-59.png
Attachment 12: Screenshot_from_2024-04-24_10-00-21.png
Screenshot_from_2024-04-24_10-00-21.png
Attachment 13: Screenshot_from_2024-04-24_10-00-37.png
Screenshot_from_2024-04-24_10-00-37.png
Attachment 14: Screenshot_from_2024-04-24_10-00-51.png
Screenshot_from_2024-04-24_10-00-51.png
Attachment 15: Screenshot_from_2024-04-24_10-01-06.png
Screenshot_from_2024-04-24_10-01-06.png
Attachment 16: Screenshot_from_2024-04-24_10-01-31.png
Screenshot_from_2024-04-24_10-01-31.png
Attachment 17: Screenshot_from_2024-04-24_10-01-39.png
Screenshot_from_2024-04-24_10-01-39.png
Attachment 18: Screenshot_from_2024-04-24_10-02-48.png
Screenshot_from_2024-04-24_10-02-48.png
Attachment 19: Screenshot_from_2024-04-24_10-03-43.png
Screenshot_from_2024-04-24_10-03-43.png
Attachment 20: Screenshot_from_2024-04-24_10-04-03.png
Screenshot_from_2024-04-24_10-04-03.png
Attachment 21: Screenshot_from_2024-04-24_10-04-24.png
Screenshot_from_2024-04-24_10-04-24.png
Attachment 22: Screenshot_from_2024-04-24_10-06-03.png
Screenshot_from_2024-04-24_10-06-03.png
Attachment 23: Screenshot_from_2024-04-24_10-07-09.png
Screenshot_from_2024-04-24_10-07-09.png
Attachment 24: Screenshot_from_2024-04-24_10-08-01.png
Screenshot_from_2024-04-24_10-08-01.png
Attachment 25: Screenshot_from_2024-04-24_10-08-56.png
Screenshot_from_2024-04-24_10-08-56.png
Attachment 26: Screenshot_from_2024-04-24_10-09-21.png
Screenshot_from_2024-04-24_10-09-21.png
Attachment 27: Screenshot_from_2024-04-24_10-10-39.png
Screenshot_from_2024-04-24_10-10-39.png
Attachment 28: Screenshot_from_2024-04-24_10-10-47.png
Screenshot_from_2024-04-24_10-10-47.png
Attachment 29: Screenshot_from_2024-04-24_10-11-34.png
Screenshot_from_2024-04-24_10-11-34.png
Attachment 30: Screenshot_from_2024-04-24_10-11-40.png
Screenshot_from_2024-04-24_10-11-40.png
Attachment 31: Screenshot_from_2024-04-24_10-36-43.png
Screenshot_from_2024-04-24_10-36-43.png
Attachment 32: Screenshot_from_2024-04-24_10-38-18.png
Screenshot_from_2024-04-24_10-38-18.png
Attachment 33: R18_131
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254316594 (  804922.0 Hz)
        Other data format:    7603408 (   24065.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        827 (       2.6 Hz)
                               RESUME:        827 (       2.6 Hz)
                              SYNC100:      33177 (     105.0 Hz)
                              WR48-63:      33177 (     105.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7535400 (   23849.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     512028 (    1620.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      315.952 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               17.612                0.000
  2                0.021                0.000
  3               36.829                0.000
  4                0.071                0.000
  5                0.375                0.000
  6                0.000                0.000
  7                0.653                0.000
  8                0.004                0.000
  9                0.000                0.000
 10                0.011                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.010                0.000
 15                0.281                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5990261      11033          0          0          0          0        778        778          0       9477          0      48652
  1   32634474     342633          0          0        250        249       4180       4180          0     333774          0      88686
  2    8233792    3266215          0          0          3          3       1445       1445          0    3263319          0      33897
  3   49889631    2758758          0          0        373        374       6684       6684          0    2744643          0      26463
  4    7710704     511890          0          0          7          7       1031       1031          0     509814          0      33167
  5   17424106     678891          0          0         57         57       2202       2202          0     674373          0      32013
  6    6276423       1654          0          0          0          0        827        827          0          0          0       9743
  7   42348474      10910          0          0         79         79       5376       5376          0          0          0     105118
  8    3355535        852          0          0          1          1        425        425          0          0          0      18387
  9    5876369       1434          0          0          0          0        717        717          0          0          0       7335
 10   19688353       5108          0          0          5          5       2549       2549          0          0          0       3256
 11    5133774       1300          0          0          1          1        649        649          0          0          0      64128
 12    6809941       1758          0          0          0          0        879        879          0          0          0       9510
 13    5093601       1298          0          0          0          0        649        649          0          0          0      11354
 14    4801692       1232          0          0          1          1        615        615          0          0          0       7416
 15   33049464       8442          0          0         50         50       4171       4171          0          0          0      12903
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.441s ( 1086.905 blocks/s,  67.932 Mb/s)
  600   Tue Apr 23 22:37:46 2024 TD00:00-08:00 Wednesday 24 April
23.34 23.4.24

      DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3

      per FEE64 rate spectra - attachments 4

      Merger etc - attachment 5

03.41 fDSSSD bias & leakage current OK - attachment 6

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 8

      per FEE64 rate spectra (off and on spill) - attachments 9-10

      Merger etc - attachment 11

03.54 analysis data file R18_67 - attachment 12
      max deadtime 11% (aida04)
      HEC data item rate c. 1.5kHz
      no timewarps

0.35 fDSSSD bias & leakage current OK - attachment 13

      FEE64 temperatures OK - attachment 14

      ADC data item stats - attachment 15

      per FEE64 rate spectra - attachments 16

      Merger etc - attachment 17

03.54 analysis data file R18_96 - attachment 18
      max deadtime 12% (aida04)
      HEC data item rate c. 1.6kHz
      no timewarps
Attachment 1: Screenshot_from_2024-04-23_23-34-18.png
Screenshot_from_2024-04-23_23-34-18.png
Attachment 2: Screenshot_from_2024-04-23_23-34-54.png
Screenshot_from_2024-04-23_23-34-54.png
Attachment 3: Screenshot_from_2024-04-23_23-35-45.png
Screenshot_from_2024-04-23_23-35-45.png
Attachment 4: Screenshot_from_2024-04-23_23-36-31.png
Screenshot_from_2024-04-23_23-36-31.png
Attachment 5: Screenshot_from_2024-04-23_23-37-21.png
Screenshot_from_2024-04-23_23-37-21.png
Attachment 6: Screenshot_from_2024-04-24_03-41-40.png
Screenshot_from_2024-04-24_03-41-40.png
Attachment 7: Screenshot_from_2024-04-24_03-42-32.png
Screenshot_from_2024-04-24_03-42-32.png
Attachment 8: Screenshot_from_2024-04-24_03-42-58.png
Screenshot_from_2024-04-24_03-42-58.png
Attachment 9: Screenshot_from_2024-04-24_03-43-35.png
Screenshot_from_2024-04-24_03-43-35.png
Attachment 10: Screenshot_from_2024-04-24_03-44-11.png
Screenshot_from_2024-04-24_03-44-11.png
Attachment 11: Screenshot_from_2024-04-24_03-44-35.png
Screenshot_from_2024-04-24_03-44-35.png
Attachment 12: R18_67
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  250287937 (  518744.3 Hz)
        Other data format:   11632065 (   24108.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1106 (       2.3 Hz)
                               RESUME:       1109 (       2.3 Hz)
                              SYNC100:      33796 (      70.0 Hz)
                              WR48-63:      33796 (      70.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   11562258 (   23963.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     752941 (    1560.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      482.488 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               23.222                0.000
  2                0.000                0.000
  3               56.625                0.000
  4                0.000                0.000
  5                0.704                0.000
  6                0.000                0.000
  7                0.939                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.006                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3375676      15359          0          0          0          0        442        442          0      14475          0      75815
  1   47916692     537101          0          0        366        367       6254       6254          0     523860          0     139250
  2    3817331    4998323          0          0          0          0       1132       1132          0    4996059          0      51826
  3   75749746    4208495          0          0        548        550      10285      10285          0    4186827          0      40621
  4    3336977     792495          0          0          1          1        545        545          0     791403          0      53742
  5   23132211    1055976          0          0         83         83       3088       3088          0    1049634          0      49833
  6    3422365        890          0          0          0          0        445        445          0          0          0      14690
  7   56668346      14890          0          0        106        106       7339       7339          0          0          0     118883
  8    2824304        672          0          0          0          0        336        336          0          0          0      28810
  9    3844587        966          0          0          0          0        483        483          0          0          0      11512
 10    4370286       1248          0          0          0          0        624        624          0          0          0       5026
 11    3088851        802          0          0          0          0        401        401          0          0          0      98441
 12    5014612       1352          0          0          0          0        676        676          0          0          0      15213
 13    3462527        896          0          0          0          0        448        448          0          0          0      18001
 14    3185286        772          0          0          2          2        384        384          0          0          0      11128
 15    7078140       1828          0          0          0          0        914        914          0          0          0      20150
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.998s ( 1032.323 blocks/s,  64.520 Mb/s)
Attachment 13: Screenshot_from_2024-04-24_07-35-18.png
Screenshot_from_2024-04-24_07-35-18.png
Attachment 14: Screenshot_from_2024-04-24_07-36-00.png
Screenshot_from_2024-04-24_07-36-00.png
Attachment 15: Screenshot_from_2024-04-24_07-36-29.png
Screenshot_from_2024-04-24_07-36-29.png
Attachment 16: Screenshot_from_2024-04-24_07-37-03.png
Screenshot_from_2024-04-24_07-37-03.png
Attachment 17: Screenshot_from_2024-04-24_07-37-45.png
Screenshot_from_2024-04-24_07-37-45.png
Attachment 18: R18_96
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  253809078 (  751826.0 Hz)
        Other data format:    8110924 (   24025.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        844 (       2.5 Hz)
                               RESUME:        844 (       2.5 Hz)
                              SYNC100:      33258 (      98.5 Hz)
                              WR48-63:      33258 (      98.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8042720 (   23823.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     529514 (    1568.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      337.590 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               19.161                0.000
  2                0.007                0.000
  3               41.205                0.000
  4                0.172                0.000
  5                0.406                0.000
  6                0.000                0.000
  7                0.656                0.000
  8                0.010                0.000
  9                0.000                0.000
 10                0.193                0.000
 11                0.001                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.023                0.000
 15                0.121                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5871650      11620          0          0          0          0        746        746          0      10128          0      52136
  1   34432367     363819          0          0        263        263       4482       4482          0     354329          0      94194
  2    7680769    3497668          0          0          2          2       1401       1401          0    3494862          0      35954
  3   53024609    2927317          0          0        398        398       7008       7008          0    2912505          0      27672
  4    8502384     548708          0          0          9          9       1159       1159          0     546372          0      36407
  5   16750131     728930          0          0         53         53       2150       2150          0     724524          0      33989
  6    6256772       1674          0          0          0          0        837        837          0          0          0      10191
  7   41893078      10740          0          0         71         71       5299       5299          0          0          0      94711
  8    3653651        920          0          0          2          2        458        458          0          0          0      20092
  9    5991221       1564          0          0          0          0        782        782          0          0          0       8149
 10   16753048       4406          0          0          5          5       2198       2198          0          0          0       3402
 11    5222175       1254          0          0          1          1        626        626          0          0          0      68583
 12    7072961       1726          0          0          0          0        863        863          0          0          0      10498
 13    5337727       1396          0          0          0          0        698        698          0          0          0      12368
 14    4940099       1278          0          0          4          4        635        635          0          0          0       7688
 15   30426436       7904          0          0         36         36       3916       3916          0          0          0      13480
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.307s ( 1055.874 blocks/s,  65.992 Mb/s)
  599   Tue Apr 23 19:23:42 2024 PP20:30 checks

All looks good.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-23_20-24-09.png
Screenshot_from_2024-04-23_20-24-09.png
Attachment 2: Screenshot_from_2024-04-23_20-24-25.png
Screenshot_from_2024-04-23_20-24-25.png
Attachment 3: Screenshot_from_2024-04-23_20-24-52.png
Screenshot_from_2024-04-23_20-24-52.png
Attachment 4: Screenshot_from_2024-04-23_20-25-35.png
Screenshot_from_2024-04-23_20-25-35.png
Attachment 5: Screenshot_from_2024-04-23_20-26-16.png
Screenshot_from_2024-04-23_20-26-16.png
Attachment 6: Screenshot_from_2024-04-23_20-27-01.png
Screenshot_from_2024-04-23_20-27-01.png
Attachment 7: Screenshot_from_2024-04-23_20-27-26.png
Screenshot_from_2024-04-23_20-27-26.png
Attachment 8: Screenshot_from_2024-04-23_20-27-53.png
Screenshot_from_2024-04-23_20-27-53.png
  598   Tue Apr 23 15:11:50 2024 TDTuesday 23 April 16:00-00:00
16.12 Analysis of data file R17_101 - attachment 1

      max deadtime c. 2% (aida02)
      no timewarps observed


16.15 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 7

      ADC data item stats - attachment 3

      per FEE64 rate spectra (on and off spill) - attachments 4-5

      Merger etc - attachment 6


16.53 aida04 stops producing data

      DAQ STOP fails for aida04
      Restart Merger to enable aida04 to complete DAQ STOP

      aida04 rebooted and DAQ restarted OK

      data file S100_alpha/R18
Attachment 1: R17_101
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  244392363 (  344543.7 Hz)
        Other data format:   17527637 (   24710.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        632 (       0.9 Hz)
                               RESUME:        632 (       0.9 Hz)
                              SYNC100:      34642 (      48.8 Hz)
                              WR48-63:      34642 (      48.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   17457089 (   24611.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     976870 (    1377.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      709.322 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1               16.367                0.000
  2                0.000                0.000
  3                3.744                0.000
  4                0.000                0.000
  5                0.028                0.000
  6                0.000                0.000
  7                0.009                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.004                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3247904      22147          0          0          0          0        434        434          0      21279          0     100052
  1   78164199     878648          0          0        419        419      10303      10303          0     857204          0     222655
  2    3679894    7126519          0          0          0          0       1427       1427          0    7123665          0      69753
  3   82858158    7034348          0          0        202        202      11907      11907          0    7010130          0      82407
  4    2430024    1045502          0          0          0          0        415        415          0    1044672          0      72940
  5   14446252    1404299          0          0          6          6       2074       2074          0    1400139          0      68087
  6    2997919        898          0          0          0          0        449        449          0          0          0      20036
  7   28730414       7980          0          0          4          4       3986       3986          0          0          0      64499
  8    2617862        676          0          0          0          0        338        338          0          0          0      38168
  9    3117608        872          0          0          0          0        436        436          0          0          0      15569
 10    3647866        954          0          0          0          0        477        477          0          0          0       7030
 11    3150764        810          0          0          0          0        405        405          0          0          0     129790
 12    4169396       1050          0          0          0          0        525        525          0          0          0      20235
 13    2725572        706          0          0          0          0        353        353          0          0          0      23658
 14    2606679        622          0          0          1          1        310        310          0          0          0      15170
 15    5801852       1606          0          0          0          0        803        803          0          0          0      26821
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.402s ( 1052.550 blocks/s,  65.784 Mb/s)
Attachment 2: Screenshot_from_2024-04-23_16-15-38.png
Screenshot_from_2024-04-23_16-15-38.png
Attachment 3: Screenshot_from_2024-04-23_16-16-52.png
Screenshot_from_2024-04-23_16-16-52.png
Attachment 4: Screenshot_from_2024-04-23_16-17-40.png
Screenshot_from_2024-04-23_16-17-40.png
Attachment 5: Screenshot_from_2024-04-23_16-17-25.png
Screenshot_from_2024-04-23_16-17-25.png
Attachment 6: Screenshot_from_2024-04-23_16-18-04.png
Screenshot_from_2024-04-23_16-18-04.png
Attachment 7: Screenshot_from_2024-04-23_16-22-04.png
Screenshot_from_2024-04-23_16-22-04.png
Attachment 8: Screenshot_from_2024-04-23_17-18-08.png
Screenshot_from_2024-04-23_17-18-08.png
Attachment 9: Screenshot_from_2024-04-23_17-18-34.png
Screenshot_from_2024-04-23_17-18-34.png
Attachment 10: Screenshot_from_2024-04-23_17-19-01.png
Screenshot_from_2024-04-23_17-19-01.png
Attachment 11: Screenshot_from_2024-04-23_17-19-18.png
Screenshot_from_2024-04-23_17-19-18.png
Attachment 12: Screenshot_from_2024-04-23_17-20-08.png
Screenshot_from_2024-04-23_17-20-08.png
Attachment 13: Screenshot_from_2024-04-23_17-20-47.png
Screenshot_from_2024-04-23_17-20-47.png
Attachment 14: Screenshot_from_2024-04-23_17-22-09.png
Screenshot_from_2024-04-23_17-22-09.png
Attachment 15: Screenshot_from_2024-04-23_17-22-09.png
Screenshot_from_2024-04-23_17-22-09.png
Attachment 16: Screenshot_from_2024-04-23_17-25-10.png
Screenshot_from_2024-04-23_17-25-10.png
Attachment 17: Screenshot_from_2024-04-23_17-29-21.png
Screenshot_from_2024-04-23_17-29-21.png
Attachment 18: Screenshot_from_2024-04-23_17-29-21.png
Screenshot_from_2024-04-23_17-29-21.png
Attachment 19: Screenshot_from_2024-04-23_17-29-40.png
Screenshot_from_2024-04-23_17-29-40.png
Attachment 20: Screenshot_from_2024-04-23_17-30-04.png
Screenshot_from_2024-04-23_17-30-04.png
  597   Tue Apr 23 13:15:43 2024 PP14:00 checks

All looks good.

Screenshots attached

Attachment 1: Screenshot_from_2024-04-23_14-11-33.png
Screenshot_from_2024-04-23_14-11-33.png
Attachment 2: Screenshot_from_2024-04-23_14-04-26.png
Screenshot_from_2024-04-23_14-04-26.png
Attachment 3: Screenshot_from_2024-04-23_14-04-44.png
Screenshot_from_2024-04-23_14-04-44.png
Attachment 4: Screenshot_from_2024-04-23_14-06-02.png
Screenshot_from_2024-04-23_14-06-02.png
Attachment 5: Screenshot_from_2024-04-23_14-07-07.png
Screenshot_from_2024-04-23_14-07-07.png
Attachment 6: Screenshot_from_2024-04-23_14-09-26.png
Screenshot_from_2024-04-23_14-09-26.png
Attachment 7: Screenshot_from_2024-04-23_14-10-55.png
Screenshot_from_2024-04-23_14-10-55.png
Attachment 8: Screenshot_from_2024-04-23_14-11-16.png
Screenshot_from_2024-04-23_14-11-16.png
  595   Tue Apr 23 13:08:28 2024 TDSummary of AIDA FEE64 stops
We have observed a number of AIDA FEE64s stop producing data.

Summary of dates & (approximate) times

aida03	21.4.24	08.20
aida04	22.4.24	09.14             https://elog.ph.ed.ac.uk/DESPEC/586
aida01	22.4.24	16.00
aida02	22.4.24	22.00
aida02	23.4.24	10.21
aida02	23.4.24 11.25
aida04  23.4.24 16.53?            attachment 2
aida04/aida05 25.4.24 22.28       no messages in aida04 system console
aida04  27.8.24 c.23:00
aida04  28.8.24 c. 12-13:00       power cycle
aida04  28.8.24 c. 16.00          reset per Elog 618
aida02  28.4.24 c. 22.20          reset per Elog 618, no error messages in system console
aida04  29.4.24 01.15
aida04  30.4.24 <03.30
aida02  30.4.24 <10.00


aida02 2.5.24 c. 19.30 reset


In future, it would be useful to copy system console reports to the Elog to check whether we can identify common error messages etc.


ssh pi@nnrpi1
<password>

cd logs
cat /ttyUSBx 

The current mapping of USB port # to AIDA FEE64s - attachment 1


aida02 system console c. 19.30 2.5.24

02:05:24/14:40:51|executing generic doStop
02:05:24/19:27:28|Halt (1)
02:05:24/19:27:29|Halt (2)
02:05:24/19:27:29|Halt (3)
02:05:24/19:27:30|Halt (4)
02:05:24/19:27:30|Halt (5)
02:05:24/19:27:31|Halt (6)
02:05:24/19:27:31|Halt (7)
02:05:24/19:27:32|Halt (8)
02:05:24/19:27:32|Halt (9)
02:05:24/19:27:33|Halt (10)
02:05:24/19:27:33|Halt (11)
02:05:24/19:27:34|Halt (12)
02:05:24/19:27:34|Halt (13)
02:05:24/19:27:35|Halt (14)
02:05:24/19:27:35|Halt (15)
02:05:24/19:27:36|Halt (16)
02:05:24/19:27:36|Halt (17)
02:05:24/19:27:37|Halt (18)
02:05:24/19:27:38|Halt (19)
02:05:24/19:27:38|Action has not completed
02:05:24/19:27:39|Giving up waiting
02:05:24/19:27:39|disconnect xfer stream #0 (1)
02:05:24/19:27:39|disconnect xfer stream #0 (2)
02:05:24/19:27:39|disconnect xfer stream #0 (3)
02:05:24/19:27:40|disconnect xfer stream #0 (4)
02:05:24/19:27:40|disconnect xfer stream #0 (5)
02:05:24/19:27:41|disconnect xfer stream #0 (6)
02:05:24/19:27:41|disconnect xfer stream #0 (7)
02:05:24/19:27:42|disconnect xfer stream #0 (8)
02:05:24/19:27:42|disconnect xfer stream #0 (9)
02:05:24/19:27:43|disconnect xfer stream #0 (10)
02:05:24/19:27:43|disconnect xfer stream #0 (11)
02:05:24/19:27:44|disconnect xfer stream #0 (12)
02:05:24/19:27:44|disconnect xfer stream #0 (13)
02:05:24/19:27:45|disconnect xfer stream #0 (14)
02:05:24/19:27:46|disconnect xfer stream #0 (15)
02:05:24/19:27:46|disconnect xfer stream #0 (16)
02:05:24/19:27:47|disconnect xfer stream #0 (17)
02:05:24/19:27:47|disconnect xfer stream #0 (18)
02:05:24/19:27:48|disconnect xfer stream #0 (19)
02:05:24/19:27:48|Action has not completed
02:05:24/19:27:49|Giving up waiting
02:05:24/19:27:49|executing generic Stop_Merger
02:05:24/19:27:49|executing generic Stop_DataStorageServer
02:05:24/19:27:49|completed generic doStop
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:27:49|do_GetState returned z=0 and 1
02:05:24/19:28:01|Transfer Error - : Connection reset by peer
02:05:24/19:28:08|send() failed: 
02:05:24/19:28:08|Aida state now Stopped. AidaExecV10.0_Jul  6 2022: Build 15:10:57 
02:05:24/19:28:08|do_GetState returned z=0 and 0
02:05:24/19:28:13|executing generic doGo
02:05:24/19:28:45|Initialising communication with storage server 192.168.11.99 on port 11002
02:05:24/19:28:45|Setting Transfer Block Size 65536
02:05:24/19:28:45|Setting Transfer Mode 3
02:05:24/19:28:45|Setting Overlap Mode 0
02:05:24/19:28:45|Setting Blocking Mode 0
02:05:24/19:28:45|Setting nice 0
02:05:24/19:28:45|TCP transfer library version 3.12
02:05:24/19:28:45|TCP socket send buffer was 16384 - now 221184
02:05:24/19:28:45|TCP socket receive buffer was 87380 - now 221184
02:05:24/19:28:45|TCP socket created OK - now connecting to 192.168.11.99 port 11002
02:05:24/19:28:45|Connected to 192.168.11.99 port 11002
02:05:24/19:28:45|connect xfer stream #0 (1)
02:05:24/19:28:45|Data Acquisition Statistics counters now cleared
02:05:24/19:28:46|ASIC DMA buffer size 2097152; WAVE DMA buffer size 1048576
02:05:24/19:28:46|Aida state now Going. AidaExecV10.0: Build Jul  6 2022_15:10:57 
02:05:24/19:28:46|Go (1)
02:05:24/19:28:46|Running Go_Electronics for aida02 with TSMaster aida01
02:05:24/19:28:46|ADCs Calibrated. 
02:05:24/19:28:46|Finished Go_Electronics for AIDA
02:05:24/19:28:46|executing generic Go_Merger
02:05:24/19:28:46|executing generic Go_DataStorageServer
02:05:24/19:28:46|completed generic doGo
02:05:24/19:28:46|do_GetState returned z=0 and 1
02:05:24/19:28:46|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
02:05:24/19:28:51|Also in RDOGo_Operate: Enabled waveform readout
02:05:24/19:28:51|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0201
02:05:24/19:28:51|WV:0:Buffer info. Length = 64 : Offset = 0 
02:05:24/19:28:51|WV:1:Buffer info. Length = 62 : Offset = 2 
02:05:24/19:28:51|WV:2:Buffer info. Length = 60 : Offset = 4 
02:05:24/19:28:52|WV:3:Buffer info. Length = 58 : Offset = 6 
02:05:24/19:28:52|WV:4:Buffer info. Length = 56 : Offset = 8 
02:05:24/19:28:52|WV:5:Buffer info. Length = 54 : Offset = 10 
02:05:24/19:28:52|WV:6:Buffer info. Length = 52 : Offset = 12 
02:05:24/19:28:52|WV:7:Buffer info. Length = 50 : Offset = 14 
02:05:24/19:28:52|WV:8:Buffer info. Length = 48 : Offset = 16 
02:05:24/19:28:52|WV:9:Buffer info. Length = 46 : Offset = 18 
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|get_WAVEBlk (A) which = 0 ---- status = 1048, State machines = 0201
02:05:24/19:28:52|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
02:05:24/19:28:52|
Attachment 1: Screenshot_from_2024-04-23_14-22-04.png
Screenshot_from_2024-04-23_14-22-04.png
Attachment 2: aida04
23:04:24/11:48:07|TCP transfer library version 3.12
23:04:24/11:48:07|TCP socket send buffer was 16384 - now 221184
23:04:24/11:48:07|TCP socket receive buffer was 87380 - now 221184
23:04:24/11:48:07|TCP socket created OK - now connecting to 192.168.11.99 port 11004
23:04:24/11:48:07|Connected to 192.168.11.99 port 11004
23:04:24/11:48:07|connect xfer stream #0 (1)
23:04:24/11:48:07|Data Acquisition Statistics counters now cleared
23:04:24/11:48:07|ASIC DMA buffer size 2097152; WAVE DMA buffer size 1048576
23:04:24/11:48:07|Aida state now Going. AidaExecV10.0: Build Jul  6 2022_15:10:57 
23:04:24/11:48:07|Go (1)
23:04:24/11:48:07|Running Go_Electronics for aida04 with TSMaster aida01
23:04:24/11:48:07|ADCs Calibrated. 
23:04:24/11:48:07|Finished Go_Electronics for AIDA
23:04:24/11:48:08|executing generic Go_Merger
23:04:24/11:48:08|executing generic Go_DataStorageServer
23:04:24/11:48:08|completed generic doGo
23:04:24/11:48:08|do_GetState returned z=0 and 1
23:04:24/11:48:08|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
23:04:24/11:48:13|Also in RDOGo_Operate: Enabled waveform readout
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 104A, State machines = 0201
23:04:24/11:48:13|WV:0:Buffer info. Length = 64 : Offset = 0 
23:04:24/11:48:13|WV:1:Buffer info. Length = 62 : Offset = 2 
23:04:24/11:48:13|WV:2:Buffer info. Length = 60 : Offset = 4 
23:04:24/11:48:13|WV:3:Buffer info. Length = 58 : Offset = 6 
23:04:24/11:48:13|WV:4:Buffer info. Length = 56 : Offset = 8 
23:04:24/11:48:13|WV:5:Buffer info. Length = 54 : Offset = 10 
23:04:24/11:48:13|WV:6:Buffer info. Length = 52 : Offset = 12 
23:04:24/11:48:13|WV:7:Buffer info. Length = 50 : Offset = 14 
23:04:24/11:48:13|WV:8:Buffer info. Length = 48 : Offset = 16 
23:04:24/11:48:13|WV:9:Buffer info. Length = 46 : Offset = 18 
23:04:24/11:48:13|get_WAVEBlk (A) which = 1 ---- status = 0188, State machines = 0201
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:13|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:13|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
23:04:24/11:48:14|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
23:04:24/11:48:14|executing generic doStop
23:04:24/16:58:30|Halt (1)
23:04:24/16:58:31|Halt (2)
23:04:24/16:58:32|Halt (3)
23:04:24/16:58:32|Halt (4)
23:04:24/16:58:33|Halt (5)
23:04:24/16:58:33|Halt (6)
23:04:24/16:58:34|Halt (7)
23:04:24/16:58:35|Halt (8)
23:04:24/16:58:35|Halt (9)
23:04:24/16:58:36|Halt (10)
23:04:24/16:58:36|Halt (11)
23:04:24/16:58:37|Halt (12)
23:04:24/16:58:37|Halt (13)
23:04:24/16:58:38|Halt (14)
23:04:24/16:58:39|Halt (15)
23:04:24/16:58:39|Halt (16)
23:04:24/16:58:40|Halt (17)
23:04:24/16:58:40|Halt (18)
23:04:24/16:58:41|Halt (19)
23:04:24/16:58:41|Action has not completed
23:04:24/16:58:42|Giving up waiting
23:04:24/16:58:42|disconnect xfer stream #0 (1)
23:04:24/16:58:42|disconnect xfer stream #0 (2)
23:04:24/16:58:43|disconnect xfer stream #0 (3)
23:04:24/16:58:43|disconnect xfer stream #0 (4)
23:04:24/16:58:44|disconnect xfer stream #0 (5)
23:04:24/16:58:44|disconnect xfer stream #0 (6)
23:04:24/16:58:45|disconnect xfer stream #0 (7)
23:04:24/16:58:45|disconnect xfer stream #0 (8)
23:04:24/16:58:46|disconnect xfer stream #0 (9)
23:04:24/16:58:47|disconnect xfer stream #0 (10)
23:04:24/16:58:47|disconnect xfer stream #0 (11)
23:04:24/16:58:48|disconnect xfer stream #0 (12)
23:04:24/16:58:48|disconnect xfer stream #0 (13)
23:04:24/16:58:49|disconnect xfer stream #0 (14)
23:04:24/16:58:49|disconnect xfer stream #0 (15)
23:04:24/16:58:50|disconnect xfer stream #0 (16)
23:04:24/16:58:51|disconnect xfer stream #0 (17)
23:04:24/16:58:51|disconnect xfer stream #0 (18)
23:04:24/16:58:52|disconnect xfer stream #0 (19)
23:04:24/16:58:52|Action has not completed
23:04:24/16:58:53|Giving up waiting
23:04:24/16:58:53|executing generic Stop_Merger
23:04:24/16:58:53|executing generic Stop_DataStorageServer
23:04:24/16:58:53|completed generic doStop
23:04:24/16:58:53|do_GetState returned z=0 and 1
23:04:24/16:58:53|do_GetState returned z=0 and 1
23:04:24/16:59:06|Transfer Error - : Connection reset by peer
23:04:24/16:59:15|send() failed: 
23:04:24/16:59:15|TCP transfer library version 3.12
23:04:24/16:59:15|TCP socket send buffer was 16384 - now 221184
23:04:24/16:59:15|TCP socket receive buffer was 87380 - now 221184
23:04:24/16:59:15|TCP socket created OK - now connecting to 192.168.11.99 port 11004
23:04:24/16:59:15|Transfer Error - : Connection refused
23:04:24/16:59:15|connect() failed:  
23:04:24/16:59:15|Aida state now Stopped. AidaExecV10.0_Jul  6 2022: Build 15:10:57 
23:04:24/16:59:15|do_GetState returned z=0 and 0
23:04:24/16:59:18|do_GetState returned z=0 and 0
23:04:24/17:00:09|do_GetState returned z=0 and 0
23:04:24/17:00:11|do_GetState returned z=0 and 0
23:04:24/17:00:19|do_GetState returned z=0 and 0
INIT: Switching to runlevel: 6
INStarting killall:  Unmounting NFS filesystems:  umount: /MIDAS: not mounted
23:04:24/17:00:43|[  OK  ]
23:04:24/17:00:43|Stopping rpcbind: [  OK  ]
23:04:24/17:00:45|Shutting down kernel logger: [  OK  ]
23:04:24/17:00:47|Shutting down system logger: [  OK  ]
23:04:24/17:00:47|Stopping xinetd: [  OK  ]
23:04:24/17:00:47|[  OK  ]
23:04:24/17:00:47|Sending all processes the TERM signal... xaida: release:
23:04:24/17:00:48|
23:04:24/17:00:48|AIDAMEM: aidamem_release:
23:04:24/17:00:48|xaida: release:
23:04:24/17:00:48|Sending all processes the KILL signal... 
23:04:24/17:00:53|Saving random seed:  
23:04:24/17:00:53|Syncing hardware clock to systxaida: open:
23:04:24/17:00:53|emxaida: release:
23:04:24/17:00:53| time xaida: open:
23:04:24/17:00:53|xaida: ioctl: called with command 536899587
23:04:24/17:00:53|xaida: ioctl: called with command 1076129801
23:04:24/17:00:53|RTCxaida: release:
23:04:24/17:00:53|_RD_TIME: Inappropriate ioctl for device
23:04:24/17:00:53|ioctl() to /dev/rtc to read the time failed.
23:04:24/17:00:53|
23:04:24/17:00:53|Please stand by while rebooting the system...
Sj5õ24/17:00:54|Restarting system.
23:04:24/17:00:56|U×®­Ë
                        )	$T,W¤']Z¥anuary 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:00:56|
Finished copying zImage to RAM 
23:04:24/17:00:57|
Found 0 errors checking kernel image
23:04:24/17:00:58|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
23:04:24/17:00:58|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
23:04:24/17:00:58|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
23:04:24/17:00:58|SPIconfig done now to set up the LMK3200 registers
23:04:24/17:00:58|LMK #0 : regInit[0]=0x80000000
23:04:24/17:00:58|LMK #0 : regInit[1]=0x10070600
23:04:24/17:00:58|LMK #0 : regInit[2]=0x60601
23:04:24/17:00:59|LMK #0 : regInit[3]=0x60602
23:04:24/17:00:59|LMK #0 : regInit[4]=0x60603
23:04:24/17:00:59|LMK #0 : regInit[5]=0x70624
23:04:24/17:00:59|LMK #0 : regInit[6]=0x70605
23:04:24/17:00:59|LMK #0 : regInit[7]=0x70606
23:04:24/17:00:59|LMK #0 : regInit[8]=0x70627
23:04:24/17:00:59|LMK #0 : regInit[9]=0x10000908
23:04:24/17:00:59|LMK #0 : regInit[10]=0xA0022A09
23:04:24/17:00:59|LMK #0 : regInit[11]=0x82800B
23:04:24/17:00:59|LMK #0 : regInit[12]=0x28C800D
23:04:24/17:00:59|LMK #0 : regInit[13]=0x830020E
23:04:24/17:00:59|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
23:04:24/17:00:59|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
23:04:24/17:00:59|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
23:04:24/17:01:00|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
23:04:24/17:01:00|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
23:04:24/17:01:03|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xa0 0x71 
23:04:24/17:01:09|Using Xilinx Virtex440 machine description
23:04:24/17:01:10|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
23:04:24/17:01:10|Zone PFN ranges:
23:04:24/17:01:10|  DMA      0x00000000 -> 0x00007000
23:04:24/17:01:10|  Normal   0x00007000 -> 0x00007000
23:04:24/17:01:10|Movable zone start PFN for each node
23:04:24/17:01:10|early_node_map[1] active PFN ranges
23:04:24/17:01:10|    0: 0x00000000 -> 0x00007000
23:04:24/17:01:10|MMU: Allocated 1088 bytes of context maps for 255 contexts
23:04:24/17:01:10|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
23:04:24/17:01:10|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
23:04:24/17:01:10|PID hash table entries: 512 (order: 9, 2048 bytes)
23:04:24/17:01:10|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
23:04:24/17:01:11|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
23:04:24/17:01:11|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
23:04:24/17:01:11|Kernel virtual memory layout:
23:04:24/17:01:11|  * 0xffffe000..0xfffff000  : fixmap
23:04:24/17:01:11|  * 0xfde00000..0xfe000000  : consistent mem
23:04:24/17:01:11|  * 0xfde00000..0xfde00000  : early ioremap
23:04:24/17:01:11|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
23:04:24/17:01:11|NR_IRQS:512
23:04:24/17:01:11|clocksource: timebase mult[a00000] shift[22] registered
23:04:24/17:01:11|Console: colour dummy device 80x25
23:04:24/17:01:11|Mount-cache hash table entries: 512
23:04:24/17:01:11|NET: Registered protocol family 16
23:04:24/17:01:11|PCI: Probing PCI hardware
23:04:24/17:01:11|bio: create slab <bio-0> at 0
23:04:24/17:01:11|NET: Registered protocol family 2
23:04:24/17:01:11|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
23:04:24/17:01:11|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
23:04:24/17:01:11|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
23:04:24/17:01:11|TCP: Hash tables configured (established 4096 bind 4096)
23:04:24/17:01:11|TCP reno registered
23:04:24/17:01:12|NET: Registered protocol family 1
23:04:24/17:01:12|ROMFS MTD (C) 2007 Red Hat, Inc.
23:04:24/17:01:12|msgmni has been set to 214
23:04:24/17:01:12|io scheduler noop registered
23:04:24/17:01:12|io scheduler anticipatory registered
23:04:24/17:01:12|io scheduler deadline registered
23:04:24/17:01:12|io scheduler cfq registered (default)
23:04:24/17:01:12|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
23:04:24/17:01:12|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
23:04:24/17:01:12|console [ttyS0] enabled
23:04:24/17:01:12|brd: module loaded
23:04:24/17:01:12|loop: module loaded
23:04:24/17:01:12|Device Tree Probing 'ethernet'
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:a0:71
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
23:04:24/17:01:12|XLlTemac: DCR address: 0x80
23:04:24/17:01:12|XLlTemac: buffer descriptor size: 32768 (0x8000)
23:04:24/17:01:12|XLlTemac: Allocating DMA descriptors with kmalloc
23:04:24/17:01:12|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
23:04:24/17:01:12|XTemac: PHY detected at address 7.
23:04:24/17:01:12|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
23:04:24/17:01:13|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13| Intel/Sharp Extended Query Table at 0x010A
23:04:24/17:01:13|Using buffer write method
23:04:24/17:01:13|cfi_cmdset_0001: Erase suspend on write enabled
23:04:24/17:01:13|cmdlinepart partition parsing not available
23:04:24/17:01:13|RedBoot partition parsing not available
23:04:24/17:01:13|Creating 5 MTD partitions on "fc000000.flash":
23:04:24/17:01:13|0x000000000000-0x000000500000 : "golden_firmware"
23:04:24/17:01:13|0x000000500000-0x000000800000 : "golden_kernel"
23:04:24/17:01:13|0x000000800000-0x000000d00000 : "user_firmware"
23:04:24/17:01:13|0x000000d00000-0x000000fe0000 : "user_kernel"
23:04:24/17:01:13|0x000000fe0000-0x000001000000 : "env_variables"
23:04:24/17:01:13|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
23:04:24/17:01:13|SPI: XIlinx spi: bus number now 32766
23:04:24/17:01:13|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
23:04:24/17:01:14|SPI: XIlinx spi: bus number now 32765
23:04:24/17:01:14|mice: PS/2 mouse device common for all mice
23:04:24/17:01:14|Device Tree Probing 'i2c'
23:04:24/17:01:14| #0 at 0x81600000 mapped to 0xD1030000, irq=22
23:04:24/17:01:14|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
23:04:24/17:01:14|TCP cubic registered
23:04:24/17:01:14|NET: Registered protocol family 17
23:04:24/17:01:14|RPC: Registered udp transport module.
23:04:24/17:01:14|RPC: Registered tcp transport module.
23:04:24/17:01:14|eth0: XLlTemac: Options: 0x3fa
23:04:24/17:01:14|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
23:04:24/17:01:14|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
23:04:24/17:01:15|eth0: XLlTemac: speed set to 1000Mb/s
23:04:24/17:01:17|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
23:04:24/17:01:17|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
23:04:24/17:01:17|Sending DHCP requests ., OK
23:04:24/17:01:18|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.4
23:04:24/17:01:18|IP-Config: Complete:
23:04:24/17:01:18|     device=eth0, addr=192.168.11.4, mask=255.255.255.0, gw=255.255.255.255,
23:04:24/17:01:18|     host=aida04, domain=dl.ac.uk, nis-domain=nuclear.physics,
23:04:24/17:01:18|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04
23:04:24/17:01:18|Looking up port of RPC 100003/2 on 192.168.11.99
23:04:24/17:01:18|Looking up port of RPC 100005/1 on 192.168.11.99
23:04:24/17:01:18|VFS: Mounted root (nfs filesystem) on device 0:12.
23:04:24/17:01:18|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
23:04:24/17:01:19|Starting sysinit...
23:04:24/17:01:19|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
23:04:24/17:01:20|		Press 'I' to enter interactive startup.
23:04:24/17:01:20|Setting clock  (utc): Tue Apr 23 16:01:20 BST 2024 [  OK  ]
23:04:24/17:01:20|Building the cache [  OK  ]
23:04:24/17:01:20|Setting hostname aida04:  [  OK  ]
23:04:24/17:01:21|Mounting local filesystems:  [  OK  ]
23:04:24/17:01:22|Enabling /etc/fstab swaps:  [  OK  ]
23:04:24/17:01:24|Finishing sysinit...
INIT: Entering runlevel: 3
... 257 more lines ...
  594   Tue Apr 23 11:09:00 2024 PP12:00 checks

Everything is OK now.

Screenshots attached.

Attachment 1: Screenshot_from_2024-04-23_12-01-49.png
Screenshot_from_2024-04-23_12-01-49.png
Attachment 2: Screenshot_from_2024-04-23_12-02-24.png
Screenshot_from_2024-04-23_12-02-24.png
Attachment 3: Screenshot_from_2024-04-23_12-03-36.png
Screenshot_from_2024-04-23_12-03-36.png
Attachment 4: Screenshot_from_2024-04-23_12-04-05.png
Screenshot_from_2024-04-23_12-04-05.png
Attachment 5: Screenshot_from_2024-04-23_12-05-52.png
Screenshot_from_2024-04-23_12-05-52.png
Attachment 6: Screenshot_from_2024-04-23_12-06-42.png
Screenshot_from_2024-04-23_12-06-42.png
Attachment 7: Screenshot_from_2024-04-23_12-07-41.png
Screenshot_from_2024-04-23_12-07-41.png
Attachment 8: Screenshot_from_2024-04-23_12-08-32.png
Screenshot_from_2024-04-23_12-08-32.png
  593   Tue Apr 23 10:52:39 2024 PPAida02 down and back

We lost aida02 at about 11:25 again. Nick fixed it about 11:50.

It is now back in the DAQ.

  592   Tue Apr 23 10:10:31 2024 PPaida02 is back

Nick rebooted AIDA and aida02 came back at around 10:55.

AIDA was out of the DAQ from about 10:30 until 10:55

  591   Tue Apr 23 09:21:01 2024 PP10:20 checks - aida02 is down

Aida02 is down. Contacted Nick.

The rest looks stable, see screenshots.

10:53 CEST (NH): Rebooted aida02 via telnet
Resync ASIC clocks
All system wide checks look ok
ASIC threshold => 0x14

Go > All 16 writing to merger, looks OK

 

aida02 went down again 11:35 CEST
Rebooted with same procedure and is back again

Attachment 1: Screenshot_from_2024-04-23_10-08-54.png
Screenshot_from_2024-04-23_10-08-54.png
Attachment 2: Screenshot_from_2024-04-23_10-13-23.png
Screenshot_from_2024-04-23_10-13-23.png
Attachment 3: Screenshot_from_2024-04-23_10-15-28.png
Screenshot_from_2024-04-23_10-15-28.png
Attachment 4: Screenshot_from_2024-04-23_10-16-23.png
Screenshot_from_2024-04-23_10-16-23.png
Attachment 5: Screenshot_from_2024-04-23_10-16-47.png
Screenshot_from_2024-04-23_10-16-47.png
Attachment 6: Screenshot_from_2024-04-23_10-17-47.png
Screenshot_from_2024-04-23_10-17-47.png
Attachment 7: Screenshot_from_2024-04-23_10-19-30.png
Screenshot_from_2024-04-23_10-19-30.png
Attachment 8: Screenshot_from_2024-04-23_10-20-20.png
Screenshot_from_2024-04-23_10-20-20.png
  590   Tue Apr 23 07:27:18 2024 PP08:00-16:00 Tuesday 24 April shift

Checks 08:00-08:30

Screenshots are attached and everything looks OK.

Attachment 1: Screenshot_from_2024-04-23_08-10-21.png
Screenshot_from_2024-04-23_08-10-21.png
Attachment 2: Screenshot_from_2024-04-23_08-13-20.png
Screenshot_from_2024-04-23_08-13-20.png
Attachment 3: Screenshot_from_2024-04-23_08-15-30.png
Screenshot_from_2024-04-23_08-15-30.png
Attachment 4: Screenshot_from_2024-04-23_08-18-28.png
Screenshot_from_2024-04-23_08-18-28.png
Attachment 5: Screenshot_from_2024-04-23_08-23-07.png
Screenshot_from_2024-04-23_08-23-07.png
Attachment 6: Screenshot_from_2024-04-23_08-26-31.png
Screenshot_from_2024-04-23_08-26-31.png
Attachment 7: Screenshot_from_2024-04-23_08-35-31.png
Screenshot_from_2024-04-23_08-35-31.png
Attachment 8: Screenshot_from_2024-04-23_08-39-43.png
Screenshot_from_2024-04-23_08-39-43.png
  589   Mon Apr 22 22:35:59 2024 TD00.00-08.00 Tuesday 22 April
23.36 Analysis of data file R17_34 - attachment 1
      max deadtime c. 1% (aida02)
      no timewarps

23.38 DSSSD bias & leakage current OK - attachment 2

      FEE64 temperatures OK - attachment 3
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 4

      per FEE64 Rate spectra - attachment 5

      Merger etc - attachment 6

01.03 DSSSD bias & leakage current OK - attachment 7

      FEE64 temperatures OK - attachment 8
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 9

      per FEE64 Rate spectra - attachment 10

      Merger etc - attachment 11

03.06 DSSSD bias & leakage current OK - attachment 12

      FEE64 temperatures OK - attachment 13
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 14

      per FEE64 Rate spectra - attachment 15

      Merger etc - attachment 16

05.04 DSSSD bias & leakage current OK - attachment 17

      FEE64 temperatures OK - attachment 18
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Merger etc - attachment 21

07.29 DSSSD bias & leakage current OK - attachment 22

      FEE64 temperatures OK - attachment 23
       aida02 ASIC temp - known fault

      ADC data item stats - attachment 24

      per FEE64 Rate spectra - attachment 25

      Merger etc - attachment 26
Attachment 1: R17_34
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  242086746 (  241481.1 Hz)
        Other data format:   19833254 (   19783.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        347 (       0.3 Hz)
                               RESUME:        347 (       0.3 Hz)
                              SYNC100:      35735 (      35.6 Hz)
                              WR48-63:      35735 (      35.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   19761090 (   19711.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     676877 (     675.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     1002.508 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                7.276                0.000
  2                0.000                0.000
  3                3.384                0.000
  4                0.000                0.000
  5                0.005                0.000
  6                0.000                0.000
  7                0.131                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.020                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    2611636      30762          0          0          0          0        342        342          0      30078          0      64523
  1   15105593     133663          0          0        117        117       1949       1949          0     129531          0      31773
  2    4737354    9053995          0          0          0          0       1811       1811          0    9050373          0      45719
  3   93081574    8973436          0          0        187        187      14170      14170          0    8944722          0      56135
  4    2149658     687339          0          0          0          0        339        339          0     686661          0      47171
  5   16270251     924233          0          0          3          3       2251       2251          0     919725          0      42612
  6    4771409       1270          0          0          0          0        635        635          0          0          0      12858
  7   64665607      18074          0          0         39         39       8998       8998          0          0          0     197755
  8    1902284        428          0          0          0          0        214        214          0          0          0      25268
  9    4387089       1362          0          0          0          0        681        681          0          0          0      10256
 10    8523582       2388          0          0          0          0       1194       1194          0          0          0       4627
 11    2287700        576          0          0          0          0        288        288          0          0          0      83305
 12    4391198       1112          0          0          0          0        556        556          0          0          0      12633
 13    3103598        860          0          0          0          0        430        430          0          0          0      15208
 14    1965500        506          0          0          0          0        253        253          0          0          0      10217
 15   12132713       3250          0          0          1          1       1624       1624          0          0          0      16817
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.539s ( 1047.838 blocks/s,  65.490 Mb/s)
Attachment 2: Screenshot_from_2024-04-22_23-38-08.png
Screenshot_from_2024-04-22_23-38-08.png
Attachment 3: Screenshot_from_2024-04-22_23-39-17.png
Screenshot_from_2024-04-22_23-39-17.png
Attachment 4: Screenshot_from_2024-04-22_23-39-44.png
Screenshot_from_2024-04-22_23-39-44.png
Attachment 5: Screenshot_from_2024-04-22_23-40-27.png
Screenshot_from_2024-04-22_23-40-27.png
Attachment 6: Screenshot_from_2024-04-22_23-40-54.png
Screenshot_from_2024-04-22_23-40-54.png
Attachment 7: Screenshot_from_2024-04-23_01-04-32.png
Screenshot_from_2024-04-23_01-04-32.png
Attachment 8: Screenshot_from_2024-04-23_01-05-40.png
Screenshot_from_2024-04-23_01-05-40.png
Attachment 9: Screenshot_from_2024-04-23_01-06-12.png
Screenshot_from_2024-04-23_01-06-12.png
Attachment 10: Screenshot_from_2024-04-23_01-06-40.png
Screenshot_from_2024-04-23_01-06-40.png
Attachment 11: Screenshot_from_2024-04-23_01-07-18.png
Screenshot_from_2024-04-23_01-07-18.png
Attachment 12: Screenshot_from_2024-04-23_03-06-28.png
Screenshot_from_2024-04-23_03-06-28.png
Attachment 13: Screenshot_from_2024-04-23_03-07-02.png
Screenshot_from_2024-04-23_03-07-02.png
Attachment 14: Screenshot_from_2024-04-23_03-07-35.png
Screenshot_from_2024-04-23_03-07-35.png
Attachment 15: Screenshot_from_2024-04-23_03-08-09.png
Screenshot_from_2024-04-23_03-08-09.png
Attachment 16: Screenshot_from_2024-04-23_03-08-43.png
Screenshot_from_2024-04-23_03-08-43.png
Attachment 17: Screenshot_from_2024-04-23_05-02-11.png
Screenshot_from_2024-04-23_05-02-11.png
Attachment 18: Screenshot_from_2024-04-23_05-03-10.png
Screenshot_from_2024-04-23_05-03-10.png
Attachment 19: Screenshot_from_2024-04-23_05-03-57.png
Screenshot_from_2024-04-23_05-03-57.png
Attachment 20: Screenshot_from_2024-04-23_05-04-37.png
Screenshot_from_2024-04-23_05-04-37.png
Attachment 21: Screenshot_from_2024-04-23_05-05-13.png
Screenshot_from_2024-04-23_05-05-13.png
Attachment 22: Screenshot_from_2024-04-23_07-29-15.png
Screenshot_from_2024-04-23_07-29-15.png
Attachment 23: Screenshot_from_2024-04-23_07-29-59.png
Screenshot_from_2024-04-23_07-29-59.png
Attachment 24: Screenshot_from_2024-04-23_07-30-27.png
Screenshot_from_2024-04-23_07-30-27.png
Attachment 25: Screenshot_from_2024-04-23_07-30-54.png
Screenshot_from_2024-04-23_07-30-54.png
Attachment 26: Screenshot_from_2024-04-23_07-31-16.png
Screenshot_from_2024-04-23_07-31-16.png
  588   Mon Apr 22 15:17:07 2024 DSJ16.00-0.00 22/04/24

First checks of shift. All looks ok.

Screenshot of HV, temps, rates, Merger attached. Attachments 1-4.

Screenshots of all spectrum Layout IDs taken before zeroing at 16.35. (Attachments 5-12)

 

checks at 16.00 - merger has crashed - aida01 dropped out  - Tom tried to stop DAQ but got an error. Restart servers restarted at 16.09 

Reset 01, restarted DAQ, and restarted as R16.

16.15 Data seems to be collected but spectra not being incremented in aida01. Timestamps look to be out of sync. Tom restarting aida01 again.

Did not fix, power cycle all FEEs - all spectra reset. aida3,6,11,12,14,15 wont calibrate adcs so wont have waveforms

RUN17 STARTED 16.51|

 HV, temps, rates, Merger looks ok 17.00. Plots saved as attachments 13-16.

 

20.47, all looks ok - see attachment 17-20. Rates are higher in attachment 20 than in attachment 2 from start of shift

21.59 aida02 (link aida1) stopped taking data and dropped out of the merger (see attachment 21). Came back to life after about 15 miutes. Tom logged in remotely to investigate. Seems to be ok. see attachment 22 for timestamps

22.48 - things apear ok. (Attachments 23-26) 

Attachment 1: Screenshot_from_2024-04-22_16-16-30.png
Screenshot_from_2024-04-22_16-16-30.png
Attachment 2: Screenshot_from_2024-04-22_16-14-31.png
Screenshot_from_2024-04-22_16-14-31.png
Attachment 3: Screenshot_from_2024-04-22_16-09-16.png
Screenshot_from_2024-04-22_16-09-16.png
Attachment 4: Screenshot_from_2024-04-22_16-05-22.png
Screenshot_from_2024-04-22_16-05-22.png
Attachment 5: Screenshot_from_2024-04-22_16-33-34.png
Screenshot_from_2024-04-22_16-33-34.png
Attachment 6: Screenshot_from_2024-04-22_16-32-54.png
Screenshot_from_2024-04-22_16-32-54.png
Attachment 7: Screenshot_from_2024-04-22_16-32-04.png
Screenshot_from_2024-04-22_16-32-04.png
Attachment 8: Screenshot_from_2024-04-22_16-31-03.png
Screenshot_from_2024-04-22_16-31-03.png
Attachment 9: Screenshot_from_2024-04-22_16-30-00.png
Screenshot_from_2024-04-22_16-30-00.png
Attachment 10: Screenshot_from_2024-04-22_16-29-03.png
Screenshot_from_2024-04-22_16-29-03.png
Attachment 11: Screenshot_from_2024-04-22_16-27-01.png
Screenshot_from_2024-04-22_16-27-01.png
Attachment 12: Screenshot_from_2024-04-22_16-25-30.png
Screenshot_from_2024-04-22_16-25-30.png
Attachment 13: Screenshot_from_2024-04-22_19-00-04.png
Screenshot_from_2024-04-22_19-00-04.png
Attachment 14: Screenshot_from_2024-04-22_18-59-25.png
Screenshot_from_2024-04-22_18-59-25.png
Attachment 15: Screenshot_from_2024-04-22_18-58-21.png
Screenshot_from_2024-04-22_18-58-21.png
Attachment 16: Screenshot_from_2024-04-22_18-56-34.png
Screenshot_from_2024-04-22_18-56-34.png
Attachment 17: Screenshot_from_2024-04-22_20-46-50.png
Screenshot_from_2024-04-22_20-46-50.png
Attachment 18: Screenshot_from_2024-04-22_20-45-57.png
Screenshot_from_2024-04-22_20-45-57.png
Attachment 19: Screenshot_from_2024-04-22_20-45-10.png
Screenshot_from_2024-04-22_20-45-10.png
Attachment 20: Screenshot_from_2024-04-22_20-44-41.png
Screenshot_from_2024-04-22_20-44-41.png
Attachment 21: Screenshot_from_2024-04-22_22-00-30.png
Screenshot_from_2024-04-22_22-00-30.png
Attachment 22: Screenshot_from_2024-04-22_22-53-11.png
Screenshot_from_2024-04-22_22-53-11.png
Attachment 23: Screenshot_from_2024-04-22_22-52-19.png
Screenshot_from_2024-04-22_22-52-19.png
Attachment 24: Screenshot_from_2024-04-22_22-50-52.png
Screenshot_from_2024-04-22_22-50-52.png
Attachment 25: Screenshot_from_2024-04-22_22-49-07.png
Screenshot_from_2024-04-22_22-49-07.png
Attachment 26: Screenshot_from_2024-04-22_22-48-31.png
Screenshot_from_2024-04-22_22-48-31.png
  587   Mon Apr 22 11:43:55 2024 TD21-22.4.24 overnight MIDASsort online sort
Attachment 1 - per DSSSD LEC e_p versus e_n (20keV/channel)

Attachment 2 - per DSSSD HEC m_p versus m_n 

Attachment 3 - per DSSSD HEC x versus y (I really do need to sort out the mapping!)

Attachment 4 - per DSSSD HEC e_p versus e_n (20MeV/channel)

Attachment 5 - per DSSSD per pixel HEC-HEC time (4.096us/channel)

Attachment 6 - per DSSSD implant/decay event rates (262.144us/channel)
               showing beam spill cycle

Attachment 7 - colours/contour levels for all 2D spectra
Attachment 1: Screenshot_from_2024-04-22_07-58-12.png
Screenshot_from_2024-04-22_07-58-12.png
Attachment 2: Screenshot_from_2024-04-22_07-59-15.png
Screenshot_from_2024-04-22_07-59-15.png
Attachment 3: Screenshot_from_2024-04-22_08-00-37.png
Screenshot_from_2024-04-22_08-00-37.png
Attachment 4: Screenshot_from_2024-04-22_08-01-08.png
Screenshot_from_2024-04-22_08-01-08.png
Attachment 5: Screenshot_from_2024-04-22_08-02-03.png
Screenshot_from_2024-04-22_08-02-03.png
Attachment 6: Screenshot_from_2024-04-22_08-03-25.png
Screenshot_from_2024-04-22_08-03-25.png
Attachment 7: Screenshot_from_2024-04-22_12-48-18.png
Screenshot_from_2024-04-22_12-48-18.png
  586   Mon Apr 22 09:03:54 2024 TDReport: aida04 stops producing data
09.14 aida04 stopped producing data - system console output - attachment 1

It is not clear (to me at least) whether the cause is radiation-induced single event upsets or
the start/stop of LN2 fills.

Nor is it clear why the FEE64 stops
- WR timestamp error (data link to Merger blocks)
- AIDAExecV10 fails
- ?

rebooted aida04

DAQ RESET/SETUP etc cycle for all FEE64s

10.03 successful restart 
      current data file R15
Attachment 1: Screenshot_from_2024-04-22_09-49-30.png
Screenshot_from_2024-04-22_09-49-30.png
  585   Sun Apr 21 23:20:45 2024 TDMonday 22 April
00.15 Data link to MBS was closed by MBS - reason(s) unknown
      NH restarted MBS receiver and data transfer restarted

01.34 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      Merger, TapeServer etc - attachment 4

04.36 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      Merger, TapeServer etc - attachment 8

07.54 data storage to local disk enabled
      file aida-gsi:/TapeData/S100_alpha/R14_

09.36 DAQ continues, current data file R14_46

      168Er setting

      DSSSD bias & leakage current OK - attachments 9 & 10
      leakage current DSSSD#1 ramping and cycling with beam spills

      FEE64 temperatures OK - attachment 11

      System wide checks OK *except* WR/FPGA errors - attachments 12-13

      WR timestamps - attachment 14
       aida04 out of seq

      ADC data item stats - attachment 15
       high rates observed all FEE64s beam on, beam off rates OK

      per FEE64 Rate spectra - attachment 16

      per FEE64 1.8.L spectra - attachments 17-18

      per FEE64 1.8.W spectra - attachments 19-20

      per FEE^4 1.8.H spectra - attachments 21-22
   
      Merger, TapeServer etc - attachments 23-25
       data rates to c. 10Mb/s beam on


09.55 aida04 rebooted - https://elog.ph.ed.ac.uk/DESPEC/586

10.01 DAQ restarted
      data file R15

      ADC data item stats - attachments 26-27
       large variation in rates beam off/on

c. 10.30 beam off (for new FRS setting?)


11.25 analysis of data files R14_19 (beam ON) and R15_24 (beam OFF) - attachments 28-29

      For beam ON max deadtime c. 11% (aida04), beam OFF max deadtime c. 0.1% (aida04)

      Zero timewarps observed

      Notice large # of PAUSE/RESUME data items for aida03 & aida04 presumably due to additional overhead of processing the time machine scalers 

12.35 Beam off

      DSSSD bias & leakage current OK - attachment 30
       Can observe beam spill structure in DSSSD leakage currents - now slowly recovering with beam off

      FEE64 temperatures OK - attachment 31

      ADC data item stats OK - attachment 32

      Merger, TapeServer etc - attachment 33

12.55 New ASIC settings 2024Apr22-12.55.38
       *all* discs disabled to reduce load on aida04
       slow comparator 0x14

      BNC PB-5 settings (since yesterday evening)

      Amplitude 10.0V
      Attenuation x10
      tau_d 1ms
      Frequency 2Hz
      Tail pulse
      Polarity +


13.46 all histograms & stats zero'd

13.48 Beam off
      per FEE64 Rate spectra - attachment 34
       2Hz pulser to p+n FEE64s

      per FEE64 1.8.W spectra - attachments 35-36
       p+n FEE64 threshold = 6000 to capture pulser events
       n+n FEE64 threshold = 8000 to capture noise

      per p+n FEE64 1.8.L spectra - attachments 37-38
       common x/y-scale - peak height proxy for peak width
       aida09 pulser peak width 57 ch FWHM ~40keV FWHM ( 5 x sigma ~ 85keV )

14.28 168Eu setting (check only - setup still in progress)
Attachment 1: Screenshot_from_2024-04-22_01-32-28.png
Screenshot_from_2024-04-22_01-32-28.png
Attachment 2: Screenshot_from_2024-04-22_01-32-48.png
Screenshot_from_2024-04-22_01-32-48.png
Attachment 3: Screenshot_from_2024-04-22_01-33-05.png
Screenshot_from_2024-04-22_01-33-05.png
Attachment 4: Screenshot_from_2024-04-22_01-34-06.png
Screenshot_from_2024-04-22_01-34-06.png
Attachment 5: Screenshot_from_2024-04-22_05-12-23.png
Screenshot_from_2024-04-22_05-12-23.png
Attachment 6: Screenshot_from_2024-04-22_05-13-16.png
Screenshot_from_2024-04-22_05-13-16.png
Attachment 7: Screenshot_from_2024-04-22_05-14-07.png
Screenshot_from_2024-04-22_05-14-07.png
Attachment 8: Screenshot_from_2024-04-22_05-15-01.png
Screenshot_from_2024-04-22_05-15-01.png
Attachment 9: Screenshot_from_2024-04-22_09-27-20.png
Screenshot_from_2024-04-22_09-27-20.png
Attachment 10: Screenshot_from_2024-04-22_09-37-32.png
Screenshot_from_2024-04-22_09-37-32.png
Attachment 11: Screenshot_from_2024-04-22_09-27-38.png
Screenshot_from_2024-04-22_09-27-38.png
Attachment 12: Screenshot_from_2024-04-22_09-27-54.png
Screenshot_from_2024-04-22_09-27-54.png
Attachment 13: Screenshot_from_2024-04-22_09-27-54.png
Screenshot_from_2024-04-22_09-27-54.png
Attachment 14: Screenshot_from_2024-04-22_09-28-29.png
Screenshot_from_2024-04-22_09-28-29.png
Attachment 15: Screenshot_from_2024-04-22_09-28-43.png
Screenshot_from_2024-04-22_09-28-43.png
Attachment 16: Screenshot_from_2024-04-22_09-29-28.png
Screenshot_from_2024-04-22_09-29-28.png
Attachment 17: Screenshot_from_2024-04-22_09-30-17.png
Screenshot_from_2024-04-22_09-30-17.png
Attachment 18: Screenshot_from_2024-04-22_09-30-48.png
Screenshot_from_2024-04-22_09-30-48.png
Attachment 19: Screenshot_from_2024-04-22_09-31-45.png
Screenshot_from_2024-04-22_09-31-45.png
Attachment 20: Screenshot_from_2024-04-22_09-32-00.png
Screenshot_from_2024-04-22_09-32-00.png
Attachment 21: Screenshot_from_2024-04-22_09-34-15.png
Screenshot_from_2024-04-22_09-34-15.png
Attachment 22: Screenshot_from_2024-04-22_09-35-55.png
Screenshot_from_2024-04-22_09-35-55.png
Attachment 23: Screenshot_from_2024-04-22_09-36-14.png
Screenshot_from_2024-04-22_09-36-14.png
Attachment 24: Screenshot_from_2024-04-22_09-36-26.png
Screenshot_from_2024-04-22_09-36-26.png
Attachment 25: Screenshot_from_2024-04-22_09-36-37.png
Screenshot_from_2024-04-22_09-36-37.png
Attachment 26: Screenshot_from_2024-04-22_10-06-16.png
Screenshot_from_2024-04-22_10-06-16.png
Attachment 27: Screenshot_from_2024-04-22_10-06-32.png
Screenshot_from_2024-04-22_10-06-32.png
Attachment 28: R14_19
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  247983505 (  854821.8 Hz)
        Other data format:   13936495 (   48040.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2020 (       7.0 Hz)
                               RESUME:       2018 (       7.0 Hz)
                              SYNC100:      33080 (     114.0 Hz)
                              WR48-63:      33080 (     114.0 Hz)
                           FEE64 disc:    4068663 (   14025.1 Hz)
                             MBS info:    9797634 (   33773.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:    1220258 (    4206.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      290.100 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.648                0.000
  1               16.396                0.000
  2               16.224                0.000
  3               32.946                0.000
  4                4.383                0.000
  5               16.546                0.000
  6                0.000                0.000
  7               17.903                0.000
  8                0.098                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.093                0.000
 12                0.128                0.000
 13                0.000                0.000
 14                6.899                0.000
 15                0.006                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   16217774     546956          0          0         43         43       2226       2226     533739       8679          0     243502
  1   35997019     655408          0          0        266        266       4741       4741     645391          3          0     273982
  2   19522166    4283326          0          0        212        211       2914       2914     609647    3667428          0     121715
  3   45252368    3863960          0          0        711        711       6334       6334     501021    3348849          0      85285
  4   13017873    1445769          0          0        136        136       1769       1769     113373    1328586          0      34444
  5   28657824    1675874          0          0        252        251       3721       3721     223840    1444089          0      62230
  6    2024666     103995          0          0          1          1        248        248     103497          0          0      18668
  7   35214745     192536          0          0        242        242       4568       4568     182916          0          0      92814
  8    9901593     241512          0          0          8          8       1198       1198     239100          0          0      66118
  9    1636292      52204          0          0          0          0        256        256      51692          0          0       9281
 10    1524839      18248          0          0          0          0        211        211      17826          0          0       2366
 11    7330756     257048          0          0         12         12        910        910     255204          0          0      83995
 12    5748840     120224          0          0         17         17        694        694     118802          0          0      35149
 13    2639782     121147          0          0          0          0        330        330     120487          0          0      30525
 14   17865155     154024          0          0        117        117       2280       2280     149230          0          0      12092
 15    5431813     204264          0          0          3          3        680        680     202898          0          0      48092
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.855s ( 1071.830 blocks/s,  66.989 Mb/s)
Attachment 29: R15_24
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  228207643 (  129261.6 Hz)
        Other data format:   33712360 (   19095.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        547 (       0.3 Hz)
                               RESUME:        547 (       0.3 Hz)
                              SYNC100:      38576 (      21.9 Hz)
                              WR48-63:      38576 (      21.9 Hz)
                           FEE64 disc:    2636338 (    1493.3 Hz)
                             MBS info:   30997776 (   17557.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     110998 (      62.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     1765.471 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                1.254                0.000
  4                0.000                0.000
  5                0.000                0.000
  6                0.000                0.000
  7                0.032                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     716460      53177          0          0          0          0        106        106          0      52965          0          0
  1    6234512      54773          0          0          0          0        905        905          1      52962          0          1
  2    1818908   15542833          0          0          0          0       2263       2263      44321   15493986          0          0
  3  117796308   17921646          0          0        542        542      20384      20384    2481931   15397863          0          3
  4     548813        118          0          0          0          0         59         59          0          0          0          0
  5    7623899       2086          0          0          0          0       1043       1043          0          0          0          0
  6    3790120       8410          0          0          0          0        635        635       7140          0          0          0
  7   58887930      93835          0          0          5          5       8712       8712      76401          0          0     110890
  8     284598       7007          0          0          0          0         30         30       6947          0          0          2
  9    5466145       1638          0          0          0          0        819        819          0          0          0          0
 10    8640524      22006          0          0          0          0       1261       1261      19484          0          0          0
 11     299711        105          0          0          0          0         40         40         25          0          0         18
 12    3513079       1030          0          0          0          0        515        515          0          0          0          0
 13    2905492        870          0          0          0          0        435        435          0          0          0          0
 14     391698        162          0          0          0          0         48         48         66          0          0         72
 15    9289446       2664          0          0          0          0       1321       1321         22          0          0         12
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.203s (  993.692 blocks/s,  62.106 Mb/s)
Attachment 30: Screenshot_from_2024-04-22_12-31-14.png
Screenshot_from_2024-04-22_12-31-14.png
Attachment 31: Screenshot_from_2024-04-22_12-31-41.png
Screenshot_from_2024-04-22_12-31-41.png
Attachment 32: Screenshot_from_2024-04-22_12-32-08.png
Screenshot_from_2024-04-22_12-32-08.png
Attachment 33: Screenshot_from_2024-04-22_12-32-45.png
Screenshot_from_2024-04-22_12-32-45.png
Attachment 34: Screenshot_from_2024-04-22_13-48-06.png
Screenshot_from_2024-04-22_13-48-06.png
Attachment 35: Screenshot_from_2024-04-22_14-07-29.png
Screenshot_from_2024-04-22_14-07-29.png
Attachment 36: Screenshot_from_2024-04-22_14-09-23.png
Screenshot_from_2024-04-22_14-09-23.png
Attachment 37: Screenshot_from_2024-04-22_14-12-50.png
Screenshot_from_2024-04-22_14-12-50.png
Attachment 38: Screenshot_from_2024-04-22_14-13-05.png
Screenshot_from_2024-04-22_14-13-05.png
  584   Sun Apr 21 00:34:33 2024 TDSunday 21 April
01.35 DSSSD bias & leakage current OK - Grafana - attachment 1

      FEE64 temperatures OK - attachment 2
 
      ADC data item stats OK - attachment 3

      Merger OK - attachment 4

04.05 DSSSD bias & leakage current OK - Grafana - attachment 5

      FEE64 temperatures OK - attachment 6
 
      ADC data item stats OK - attachment 7

      Merger OK - attachment 8

08.20 DSSSD bias & leakage current OK - Grafana - attachment 9

      FEE64 temperatures OK - attachment 10
 
      ADC data item stats - attachment 11
       increased rates observed esp. DSSSD#2 n+n

      Merger etc - attachment 12
      rate increased x2 cf. overnight

      WR timestamp - attachment 13
       aida03 out sequence

09.25 MBS data relay restarted

      Merger data links - attachment 14
       aida03 rate zero

      per FEE64 1.8.W spectra - 20us FSR - attachments 15-16

      per FEE64 Rate spectra - attachment 17
      increased rate for DSSSD#2 n+n FEE64s in particular

      MIDASsort online analysis - attachments 18-19
      per DSSSD e_front versus e_back for LEC (20keV/channel) and HEC (20MeV/channel)
      observe light ions (LEC) c. 2MeV
      observe low rate of high energy ions (HEC) c. 600MeV
      similar energies in both DSSSDs for LEC and HEC implying both types of event have range >> 2mm Si


10.09 DAQ STOP
      aida03 does not respond - reboot aida03

      DAQ RESET/SETUP etc     

      Synchronise ASIC clocks
      All system wide checks OK *except* FPGA errors

      Data transfer disabled
      Histograms & stats zero'd
      DAQ GO

      Merger restarted 

      Data transfer enabled
      DAQ GO

      WR timestamp OK - attachments 20-21

      ADC data item stats OK - attachment 22
       rates status quo ante
  
      Merger etc OK - attachment 23


14.58 DSSSD bias & leakage current OK - Grafana - attachment 24

      FEE64 temperatures OK - attachment 25

      ADC data item stats OK - attachment 26

      Merger etc OK - attachment 27

18.13 DSSSD bias & leakage current OK - Grafana - attachment 28

      FEE64 temperatures OK - attachment 29

      ADC data item stats OK - attachment 30

      Merger etc OK - attachment 31


20.30 Primary c, few kHz 170Er beam to S4 - upstream s/steel flange removed
      Start of stopping range calibration in the implantation stack - currently ion appear to be stopping in DSSSD#1 (upstream)


      Decay and implant events as function of time showing spill structure - attachment 32

      per DSSSD e_front versus e_back (HEC, 20MeV/channel) - attachment 33

      per DSSSD x versus y hit pattern (HEC) - attachment 34
       mapping to be configured but clear ions are mostly implanted in centre of DSSSD#1 

      2D colour/contour levels - attachment 35

      per DSSSD m_front versus m_back (HEC) - attachment 36

      per DSSSD per pixel HEC-HEC time (4.096us/channel) - attachment 37

23.44 aida02 has stopped running
      DAQ STOP
      aida02 rebooted

      Data transfer disabled

      DAQ RESET/SETUP 

      Synchronise ASIC clocks
      All system wide checks OK *except* FPGA errors

      Histograms & stats zero'd
      DAQ GO

      Merger restarted 

      Data transfer enabled
      DAQ GO







        


        
Attachment 1: Screenshot_from_2024-04-21_01-31-59.png
Screenshot_from_2024-04-21_01-31-59.png
Attachment 2: Screenshot_from_2024-04-21_01-32-50.png
Screenshot_from_2024-04-21_01-32-50.png
Attachment 3: Screenshot_from_2024-04-21_01-33-28.png
Screenshot_from_2024-04-21_01-33-28.png
Attachment 4: Screenshot_from_2024-04-21_01-34-10.png
Screenshot_from_2024-04-21_01-34-10.png
Attachment 5: Screenshot_from_2024-04-21_04-05-57.png
Screenshot_from_2024-04-21_04-05-57.png
Attachment 6: Screenshot_from_2024-04-21_04-06-58.png
Screenshot_from_2024-04-21_04-06-58.png
Attachment 7: Screenshot_from_2024-04-21_04-07-36.png
Screenshot_from_2024-04-21_04-07-36.png
Attachment 8: Screenshot_from_2024-04-21_04-08-08.png
Screenshot_from_2024-04-21_04-08-08.png
Attachment 9: Screenshot_from_2024-04-21_08-18-51.png
Screenshot_from_2024-04-21_08-18-51.png
Attachment 10: Screenshot_from_2024-04-21_08-19-50.png
Screenshot_from_2024-04-21_08-19-50.png
Attachment 11: Screenshot_from_2024-04-21_08-21-20.png
Screenshot_from_2024-04-21_08-21-20.png
Attachment 12: Screenshot_from_2024-04-21_08-22-42.png
Screenshot_from_2024-04-21_08-22-42.png
Attachment 13: Screenshot_from_2024-04-21_08-28-22.png
Screenshot_from_2024-04-21_08-28-22.png
Attachment 14: Screenshot_from_2024-04-21_09-27-50.png
Screenshot_from_2024-04-21_09-27-50.png
Attachment 15: Screenshot_from_2024-04-21_09-29-07.png
Screenshot_from_2024-04-21_09-29-07.png
Attachment 16: Screenshot_from_2024-04-21_09-30-08.png
Screenshot_from_2024-04-21_09-30-08.png
Attachment 17: Screenshot_from_2024-04-21_09-31-12.png
Screenshot_from_2024-04-21_09-31-12.png
Attachment 18: Screenshot_from_2024-04-21_09-37-11.png
Screenshot_from_2024-04-21_09-37-11.png
Attachment 19: Screenshot_from_2024-04-21_09-38-08.png
Screenshot_from_2024-04-21_09-38-08.png
Attachment 20: Screenshot_from_2024-04-21_10-18-25.png
Screenshot_from_2024-04-21_10-18-25.png
Attachment 21: Screenshot_from_2024-04-21_10-32-41.png
Screenshot_from_2024-04-21_10-32-41.png
Attachment 22: Screenshot_from_2024-04-21_10-33-08.png
Screenshot_from_2024-04-21_10-33-08.png
Attachment 23: Screenshot_from_2024-04-21_10-33-33.png
Screenshot_from_2024-04-21_10-33-33.png
Attachment 24: Screenshot_from_2024-04-21_14-58-07.png
Screenshot_from_2024-04-21_14-58-07.png
Attachment 25: Screenshot_from_2024-04-21_14-58-31.png
Screenshot_from_2024-04-21_14-58-31.png
Attachment 26: Screenshot_from_2024-04-21_14-58-48.png
Screenshot_from_2024-04-21_14-58-48.png
Attachment 27: Screenshot_from_2024-04-21_14-59-00.png
Screenshot_from_2024-04-21_14-59-00.png
Attachment 28: Screenshot_from_2024-04-21_18-13-44.png
Screenshot_from_2024-04-21_18-13-44.png
Attachment 29: Screenshot_from_2024-04-21_18-14-06.png
Screenshot_from_2024-04-21_18-14-06.png
Attachment 30: Screenshot_from_2024-04-21_18-14-24.png
Screenshot_from_2024-04-21_18-14-24.png
Attachment 31: Screenshot_from_2024-04-21_18-14-40.png
Screenshot_from_2024-04-21_18-14-40.png
Attachment 32: Screenshot_from_2024-04-21_20-54-50.png
Screenshot_from_2024-04-21_20-54-50.png
Attachment 33: Screenshot_from_2024-04-21_20-54-18.png
Screenshot_from_2024-04-21_20-54-18.png
Attachment 34: Screenshot_from_2024-04-21_20-54-03.png
Screenshot_from_2024-04-21_20-54-03.png
Attachment 35: Screenshot_from_2024-04-21_20-53-41.png
Screenshot_from_2024-04-21_20-53-41.png
Attachment 36: Screenshot_from_2024-04-21_20-53-31.png
Screenshot_from_2024-04-21_20-53-31.png
Attachment 37: Screenshot_from_2024-04-21_21-02-02.png
Screenshot_from_2024-04-21_21-02-02.png
  583   Sat Apr 20 17:52:27 2024 TD, NHSaturday 20 April contd.
Note 40kBq 60Co source placed on bottom surface of snout centred on bPlas detectors and DSSSDs

18.50 Primary beam 750MeV/u Er to S4
      Primary beam stopped by flange (5cm s/steel in position upstream of implantation stack)

      Observe c. 2MeV dE peak in AIDA DSSSD#1 and DSSSD#2 using MIDASsort online analysis - attachment 1



21.50 Primary beam 1000MeV/u Er to S4 - attachment 2

      per DSSSD e_front versus e_back, 20keV/channel
      DSSSD #1 x & y -axis projections

10.50 Overnight stats - attachment 3
      Primary beam ?
      per DSSSD e_front versus e_back, 20keV/channel
      DSSSD #1 x & y -axis projections
      note multiple dE peaks observed in both DSSSDs
Attachment 1: Screenshot_from_2024-04-20_18-51-53.png
Screenshot_from_2024-04-20_18-51-53.png
Attachment 2: Screenshot_from_2024-04-20_21-52-08.png
Screenshot_from_2024-04-20_21-52-08.png
Attachment 3: Screenshot_from_2024-04-21_10-52-19.png
Screenshot_from_2024-04-21_10-52-19.png
  582   Sat Apr 20 11:40:38 2024 TDOffline analysis R12 & R13 ( pulser walkthrough )
S100 ADC offsets using pulser walkthrough data from data files R12 & R13

ch = channel + ( module * 64 ) + ( range * 2048 )

adc_data( ch ) = INT( RSHIFT( ABS( adc_data( ch ) - 32768 ), 3 ) - offset( ch ) + 0.5 )




per p+n FEE64 ADC spectra (aida01-aida16 *not* aida02, aida04, aida06 & aida08) - attachments 1-12

Note common x & y-scales - peak height proxy for peak width

aida09 1.8.L (s520) pulser peak width 7 ch FWHM (~40keV FWHM)




per n+n FEE64 ADC spectra (aida02, aida04, aida06 & aida08) - attachments 13-16

Note common x & y-scales - peak height proxy for peak width

aida02 1.8.L (s72) pulser peak width 16 ch FWHM (~88keV FWHM)




ADC offset analysis

/home/npg/td/Calibration/offsets /home/npg/td/S100/R12-R13 50 1024 20 5


First pass sort variables ( ADC offsets, FEE64 configuration, LEC & HEC equal energy cuts ) - attachment 17
Attachment 1: Screenshot_from_2024-04-20_12-36-14.png
Screenshot_from_2024-04-20_12-36-14.png
Attachment 2: Screenshot_from_2024-04-20_12-36-35.png
Screenshot_from_2024-04-20_12-36-35.png
Attachment 3: Screenshot_from_2024-04-20_12-36-55.png
Screenshot_from_2024-04-20_12-36-55.png
Attachment 4: Screenshot_from_2024-04-20_12-37-21.png
Screenshot_from_2024-04-20_12-37-21.png
Attachment 5: Screenshot_from_2024-04-20_12-37-55.png
Screenshot_from_2024-04-20_12-37-55.png
Attachment 6: Screenshot_from_2024-04-20_12-38-22.png
Screenshot_from_2024-04-20_12-38-22.png
Attachment 7: Screenshot_from_2024-04-20_12-38-47.png
Screenshot_from_2024-04-20_12-38-47.png
Attachment 8: Screenshot_from_2024-04-20_12-39-05.png
Screenshot_from_2024-04-20_12-39-05.png
Attachment 9: Screenshot_from_2024-04-20_12-39-21.png
Screenshot_from_2024-04-20_12-39-21.png
Attachment 10: Screenshot_from_2024-04-20_12-39-34.png
Screenshot_from_2024-04-20_12-39-34.png
Attachment 11: Screenshot_from_2024-04-20_12-39-52.png
Screenshot_from_2024-04-20_12-39-52.png
Attachment 12: Screenshot_from_2024-04-20_12-40-17.png
Screenshot_from_2024-04-20_12-40-17.png
Attachment 13: Screenshot_from_2024-04-20_12-51-16.png
Screenshot_from_2024-04-20_12-51-16.png
Attachment 14: Screenshot_from_2024-04-20_12-51-38.png
Screenshot_from_2024-04-20_12-51-38.png
Attachment 15: Screenshot_from_2024-04-20_12-52-00.png
Screenshot_from_2024-04-20_12-52-00.png
Attachment 16: Screenshot_from_2024-04-20_12-52-53.png
Screenshot_from_2024-04-20_12-52-53.png
Attachment 17: variables.dat
  $variables
  offset(   0) =    34.20
  offset(   1) =    27.50
  offset(   2) =   -18.51
  offset(   3) =    13.56
  offset(   4) =    -6.12
  offset(   5) =   -14.51
  offset(   6) =    11.86
  offset(   7) =    -0.19
  offset(   8) =     2.52
  offset(   9) =    24.94
  offset(  10) =     7.54
  offset(  11) =   -20.01
  offset(  12) =    -0.93
  offset(  13) =    26.51
  offset(  14) =   -12.75
  offset(  15) =    -4.72
  offset(  16) =     5.48
  offset(  17) =    -8.97
  offset(  18) =     4.25
  offset(  19) =    -3.19
  offset(  20) =   -40.78
  offset(  21) =    -0.95
  offset(  22) =   -14.62
  offset(  23) =   -28.18
  offset(  24) =   -12.96
  offset(  25) =    14.90
  offset(  26) =   -32.39
  offset(  27) =     0.66
  offset(  28) =     6.68
  offset(  29) = -9999.99
  offset(  30) =    15.03
  offset(  31) =    -2.98
  offset(  32) =    16.22
  offset(  33) =    27.02
  offset(  34) = -9999.99
  offset(  35) =    17.09
  offset(  36) =    17.72
  offset(  37) =    16.29
  offset(  38) =    23.46
  offset(  39) =    27.55
  offset(  40) =     9.26
  offset(  41) =    34.03
  offset(  42) =     2.46
  offset(  43) =     8.62
  offset(  44) =    29.61
  offset(  45) =   -14.47
  offset(  46) =    42.23
  offset(  47) =   -27.52
  offset(  48) =     1.06
  offset(  49) =     4.03
  offset(  50) =    22.04
  offset(  51) =   -10.44
  offset(  52) =    38.54
  offset(  53) =     8.66
  offset(  54) =    20.65
  offset(  55) =     0.35
  offset(  56) =     5.19
  offset(  57) =    19.55
  offset(  58) =    -5.57
  offset(  59) =    -7.26
  offset(  60) =   -21.96
  offset(  61) =     5.89
  offset(  62) =    36.14
  offset(  63) =     3.05
  offset(  64) =   -15.80
  offset(  65) =    25.02
  offset(  66) =   -15.94
  offset(  67) =     6.19
  offset(  68) =   -19.87
  offset(  69) =     0.37
  offset(  70) =    -9.68
  offset(  71) =     9.68
  offset(  72) =   -11.13
  offset(  73) =    -6.20
  offset(  74) =   -25.35
  offset(  75) =   -25.57
  offset(  76) =   -19.75
  offset(  77) =     8.60
  offset(  78) =   -22.55
  offset(  79) =   -10.56
  offset(  80) =     5.90
  offset(  81) =    -1.29
  offset(  82) =    -1.52
  offset(  83) = -9999.99
  offset(  84) =    24.56
  offset(  85) =   -10.73
  offset(  86) =    11.96
  offset(  87) =   -35.20
  offset(  88) =    16.67
  offset(  89) =    10.31
  offset(  90) =   -13.86
  offset(  91) =   -33.64
  offset(  92) =     2.96
  offset(  93) =    -2.41
  offset(  94) =     1.80
  offset(  95) =    26.72
  offset(  96) =   -18.06
  offset(  97) =    -0.82
  offset(  98) =   -19.61
  offset(  99) = -9999.99
  offset( 100) =   -11.22
  offset( 101) =     9.13
  offset( 102) =   -29.56
  offset( 103) =    -1.95
  offset( 104) =   -14.24
  offset( 105) =   -21.87
  offset( 106) =    -7.47
  offset( 107) =   -14.22
  offset( 108) =    -6.72
  offset( 109) =    -4.35
  offset( 110) =   -12.98
  offset( 111) =   -11.42
  offset( 112) =     5.39
  offset( 113) =    16.43
  offset( 114) =   -10.19
  offset( 115) =   -13.21
  offset( 116) =    19.82
  offset( 117) =    -4.45
  offset( 118) =    21.88
  offset( 119) =    13.84
  offset( 120) =    10.16
  offset( 121) = -9999.99
  offset( 122) = -9999.99
  offset( 123) = -9999.99
  offset( 124) =    13.42
  offset( 125) =     4.74
  offset( 126) =    10.81
  offset( 127) =    16.15
  offset( 128) =    -4.37
  offset( 129) =    21.60
  offset( 130) =     8.61
  offset( 131) =    -2.95
  offset( 132) =   -10.00
  offset( 133) =     2.40
  offset( 134) =    -9.40
  offset( 135) =   -24.68
  offset( 136) =     4.46
  offset( 137) =    -0.20
  offset( 138) =    20.63
  offset( 139) =   -25.14
  offset( 140) =    -7.48
  offset( 141) =    13.71
  offset( 142) =    -4.11
  offset( 143) =    -9.17
  offset( 144) =    15.36
  offset( 145) =    11.07
  offset( 146) =   -13.21
  offset( 147) =    -8.91
  offset( 148) =     4.75
  offset( 149) =   -10.47
  offset( 150) =    18.10
  offset( 151) =     0.62
  offset( 152) =    -2.18
  offset( 153) =     5.79
  offset( 154) =     2.73
  offset( 155) =    17.67
  offset( 156) =    15.58
  offset( 157) =   -34.45
  offset( 158) =    -1.55
  offset( 159) =    12.93
  offset( 160) =     4.16
  offset( 161) =   -15.55
  offset( 162) =    17.51
  offset( 163) =    -5.30
  offset( 164) =     8.65
  offset( 165) =   -13.48
  offset( 166) =   -10.14
  offset( 167) =    -3.75
  offset( 168) =    12.28
  offset( 169) =   -23.31
  offset( 170) =   -11.44
  offset( 171) =     5.06
  offset( 172) =   -15.18
  offset( 173) =    17.88
  offset( 174) =    12.26
  offset( 175) =   -13.64
  offset( 176) =    -9.33
  offset( 177) =    21.57
  offset( 178) =    28.82
  offset( 179) =    27.67
  offset( 180) =    62.22
  offset( 181) =    -2.89
  offset( 182) =    17.11
  offset( 183) =    11.80
  offset( 184) =    17.26
  offset( 185) =    15.52
  offset( 186) =    -8.89
  offset( 187) =    15.53
  offset( 188) =   -11.80
  offset( 189) =     0.85
  offset( 190) =    11.21
  offset( 191) =     7.76
  offset( 192) =     5.12
  offset( 193) =    26.29
  offset( 194) = -9999.99
  offset( 195) =     7.45
  offset( 196) = -9999.99
  offset( 197) =     2.31
  offset( 198) =    -1.94
  offset( 199) = -9999.99
  offset( 200) =    -0.41
  offset( 201) =   -23.18
  offset( 202) =    -1.82
  offset( 203) =    17.39
  offset( 204) =   -10.40
  offset( 205) =   -13.42
  offset( 206) =     8.22
  offset( 207) =     2.93
  offset( 208) =   -56.48
  offset( 209) =   -10.42
  offset( 210) =    50.72
  offset( 211) =     0.45
  offset( 212) =    27.74
  offset( 213) =   -16.62
  offset( 214) =    15.39
  offset( 215) =   -13.44
  offset( 216) =    34.58
  offset( 217) =    15.20
  offset( 218) =    -3.05
  offset( 219) =    -0.28
  offset( 220) =   -40.52
  offset( 221) =   -31.47
  offset( 222) =    20.29
  offset( 223) =   -13.69
  offset( 224) =    -1.11
  offset( 225) =    33.58
  offset( 226) =   -16.57
  offset( 227) =    -4.95
  offset( 228) =   -11.93
  offset( 229) =     7.06
  offset( 230) =    -2.90
  offset( 231) =   -38.90
  offset( 232) =    -8.84
  offset( 233) =     1.20
  offset( 234) =     3.21
  offset( 235) =    17.26
  offset( 236) =     4.76
  offset( 237) =    13.15
  offset( 238) =    13.66
  offset( 239) =    -2.42
  offset( 240) =    -1.46
  offset( 241) =     0.03
  offset( 242) =    20.72
  offset( 243) =    -0.74
  offset( 244) =    -2.03
  offset( 245) =   -36.41
  offset( 246) =     5.31
  offset( 247) =     3.45
  offset( 248) =    -9.35
  offset( 249) =     8.61
  offset( 250) =     6.18
  offset( 251) =   -72.01
  offset( 252) =    11.53
  offset( 253) =   -13.74
  offset( 254) =    -2.41
  offset( 255) =   -16.47
  offset( 256) =     6.19
  offset( 257) =    13.35
  offset( 258) =    10.44
  offset( 259) =    16.45
  offset( 260) =    13.62
  offset( 261) =    36.59
  offset( 262) =    24.89
  offset( 263) =    11.04
  offset( 264) =    12.12
  offset( 265) =    -1.04
  offset( 266) =    20.55
  offset( 267) =   -19.88
  offset( 268) =    10.57
  offset( 269) =    31.05
  offset( 270) =    23.09
  offset( 271) =   -17.38
  offset( 272) =   -18.40
  offset( 273) =    16.00
  offset( 274) =    -5.59
  offset( 275) =    -3.98
  offset( 276) =     0.70
  offset( 277) =     4.32
  offset( 278) =   -13.61
  offset( 279) =    26.62
  offset( 280) =    17.81
  offset( 281) =     7.10
  offset( 282) =   -30.97
  offset( 283) =    13.50
  offset( 284) =    88.96
  offset( 285) =    -0.03
  offset( 286) =     8.64
  offset( 287) =     1.86
  offset( 288) =     4.27
  offset( 289) =    19.21
  offset( 290) =   -33.21
  offset( 291) =   -28.54
  offset( 292) =     7.98
  offset( 293) =    -8.69
  offset( 294) =     1.27
  offset( 295) =    20.08
  offset( 296) =    15.92
  offset( 297) =    -4.67
  offset( 298) =    30.68
... 1771 more lines ...
  581   Sat Apr 20 03:20:23 2024 TDSaturday 20 April
Note 60Co source (current activity c. 43kBq) was installed yesterday evening - assume it is positioned on the bottom side of the snout as per the other sources used this week/

04.17 DSSSD bias & leakage current OK - Grafana - attachment 1

      FEE64 temperatures OK - attachment 2
       aida02 ASIC temp known to be u/s

      ADC data item stats - attachment 3
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs

      per FEE64 Rate spectra - attachment 4

      Merger etc - attachment 5


11.24 DSSSD bias & leakage current OK - Grafana - attachment 6

      FEE64 temperatures OK - attachment 7
       aida02 ASIC temp known to be u/s

      ADC data item stats - attachment 8
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs

      per FEE64 Rate spectra - attachment 9

      Merger etc - attachment 10


17.50 DSSSD bias & leakage current OK - Grafana - attachment 11

      FEE64 temperatures OK - attachment 12
       aida02 ASIC temp known to be u/s

      ALl syhstem wide checks OK *except* WR/FPGA errors - attachments 13-14

      WR timestamps OK - attachment 15

      per FEE64 Rate spectra - attachment 16

      per FEE64 1.8.W spectra - 20us FSR - attachments 17-18

      Merger, TapeServer, Data Links - attachments 19-20
       no data storage

      ADC data item stats - attachment 21
       all FEE64 slow comparators 0x14 from yesterday's pulser walkthroughs


18.01 Grafana alerting shows detail of DSSSD leakage current and S4 temperature variations over time - attachment  22
       DEGAS LN2 fill cycle every 6h 10.00 - 16.00 - 22.00 ... etc

https://despec-vm-01.gsi.de/grafana/d/tHRVJYfGz/aida-alerting?orgId=1
Attachment 1: Screenshot_from_2024-04-20_04-17-19.png
Screenshot_from_2024-04-20_04-17-19.png
Attachment 2: Screenshot_from_2024-04-20_04-17-52.png
Screenshot_from_2024-04-20_04-17-52.png
Attachment 3: Screenshot_from_2024-04-20_04-18-26.png
Screenshot_from_2024-04-20_04-18-26.png
Attachment 4: Screenshot_from_2024-04-20_04-19-13.png
Screenshot_from_2024-04-20_04-19-13.png
Attachment 5: Screenshot_from_2024-04-20_04-19-58.png
Screenshot_from_2024-04-20_04-19-58.png
Attachment 6: Screenshot_from_2024-04-20_11-25-02.png
Screenshot_from_2024-04-20_11-25-02.png
Attachment 7: Screenshot_from_2024-04-20_11-25-54.png
Screenshot_from_2024-04-20_11-25-54.png
Attachment 8: Screenshot_from_2024-04-20_11-26-43.png
Screenshot_from_2024-04-20_11-26-43.png
Attachment 9: Screenshot_from_2024-04-20_11-27-20.png
Screenshot_from_2024-04-20_11-27-20.png
Attachment 10: Screenshot_from_2024-04-20_11-28-05.png
Screenshot_from_2024-04-20_11-28-05.png
Attachment 11: Screenshot_from_2024-04-20_17-50-12.png
Screenshot_from_2024-04-20_17-50-12.png
Attachment 12: Screenshot_from_2024-04-20_17-50-38.png
Screenshot_from_2024-04-20_17-50-38.png
Attachment 13: Screenshot_from_2024-04-20_17-51-20.png
Screenshot_from_2024-04-20_17-51-20.png
Attachment 14: Screenshot_from_2024-04-20_17-51-03.png
Screenshot_from_2024-04-20_17-51-03.png
Attachment 15: Screenshot_from_2024-04-20_17-54-41.png
Screenshot_from_2024-04-20_17-54-41.png
Attachment 16: Screenshot_from_2024-04-20_17-51-58.png
Screenshot_from_2024-04-20_17-51-58.png
Attachment 17: Screenshot_from_2024-04-20_17-52-58.png
Screenshot_from_2024-04-20_17-52-58.png
Attachment 18: Screenshot_from_2024-04-20_17-52-26.png
Screenshot_from_2024-04-20_17-52-26.png
Attachment 19: Screenshot_from_2024-04-20_17-53-32.png
Screenshot_from_2024-04-20_17-53-32.png
Attachment 20: Screenshot_from_2024-04-20_17-53-24.png
Screenshot_from_2024-04-20_17-53-24.png
Attachment 21: Screenshot_from_2024-04-20_17-53-15.png
Screenshot_from_2024-04-20_17-53-15.png
Attachment 22: Screenshot_from_2024-04-20_17-51-37.png
Screenshot_from_2024-04-20_17-51-37.png
Attachment 23: Screenshot_from_2024-04-20_18-05-01.png
Screenshot_from_2024-04-20_18-05-01.png
  580   Fri Apr 19 13:52:34 2024 JB, CC, TDFriday 19 April contd.
12.00 Replaced AIDA ASIC mezzanine of aida01 to fix issue with asic #3

      During replacement the HDMI connector of the adjacent FEE64 aida09 became disconnected from the FEE64 PCB
      aida09 ( MAC ee:10 ) replaced (MAC 41:cf:ad )

      AIDA FEE64 adaptor PCBs for aida01, aida14 and aida09 disconnected and re-connected during this process 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
       *except* aida02 ASIC temp - known fault

      System wide checks OK *except* WR/FPGA errors - attachments 3-4

      WR timestamps OK - attachment 5

      ADC, DISC, PAUSE, RESUME and correlation data stats - attachments 6-10
       ADC data items 10/16 < 20k, max 143k

      per FEE64 Rate spectra - attachment 11

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-13


15.10 Synchronise ASIC clocks
      Re-calibrate ALL ADCs 

      ASIC settings 2024Mar27-11.25.32

      Changes ( https://elog.ph.ed.ac.uk/CARME/499 )

       IBias LF feedback from 0xf to 0x8
       Diode link threshold from 0xbf to 0xca

      New ASIC settings saved as 2024Apr19-15.22.49

      All FEE64 slow comparators -> 0x14

      Data file S100_alpha/R12

      Pulser ealkthrough (test +)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity +
      Tail pulse
      tau_d 1ms

16.35 DAQ ends OK file S100_alpha/R12_24

      per p+n FEE64 1.8.L spectra - attachment 14
       aida09 pulser peak width 54 ch FWHM 

17.38 Data file S100_alpha/R13

      Pulser walkthrough (test -)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity -
      Tail pulse
      tau_d 1ms

18.43 DAQ ends OK file S100_alpha/R13_20

      per p+n FEE64 1.8.L spectra - attachment 15

18.55 Current status

      FEE64 power ON
      DSSSD bias ON

      All FEE64 slow comparator 0x14

      DAQ going -> Merger -> TapeServer (no storage mode) -> MBS ( but data stream not yet being read by MBS )


To Do

- S4 currently closed/controlled access for FRS startup

If we have access to S4 tomorrow

- switch test - to test + ( not critical )
- test AIDA interlock 
- further alpha background?



22.47 DSSSD bias & leakage current OK - attachments 16-17

      FEE64 temps OK - attachment 18
       *except* aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Data link, Tape Server & Merger - attachments 21-23

       
Attachment 1: Screenshot_from_2024-04-19_14-56-51.png
Screenshot_from_2024-04-19_14-56-51.png
Attachment 2: Screenshot_from_2024-04-19_14-57-04.png
Screenshot_from_2024-04-19_14-57-04.png
Attachment 3: Screenshot_from_2024-04-19_14-58-27.png
Screenshot_from_2024-04-19_14-58-27.png
Attachment 4: Screenshot_from_2024-04-19_14-58-23.png
Screenshot_from_2024-04-19_14-58-23.png
Attachment 5: Screenshot_from_2024-04-19_14-59-02.png
Screenshot_from_2024-04-19_14-59-02.png
Attachment 6: Screenshot_from_2024-04-19_15-01-31.png
Screenshot_from_2024-04-19_15-01-31.png
Attachment 7: Screenshot_from_2024-04-19_15-01-17.png
Screenshot_from_2024-04-19_15-01-17.png
Attachment 8: Screenshot_from_2024-04-19_15-01-02.png
Screenshot_from_2024-04-19_15-01-02.png
Attachment 9: Screenshot_from_2024-04-19_15-00-39.png
Screenshot_from_2024-04-19_15-00-39.png
Attachment 10: Screenshot_from_2024-04-19_14-59-51.png
Screenshot_from_2024-04-19_14-59-51.png
Attachment 11: Screenshot_from_2024-04-19_15-02-09.png
Screenshot_from_2024-04-19_15-02-09.png
Attachment 12: Screenshot_from_2024-04-19_15-04-36.png
Screenshot_from_2024-04-19_15-04-36.png
Attachment 13: Screenshot_from_2024-04-19_15-03-48.png
Screenshot_from_2024-04-19_15-03-48.png
Attachment 14: Screenshot_from_2024-04-19_16-56-36.png
Screenshot_from_2024-04-19_16-56-36.png
Attachment 15: Screenshot_from_2024-04-19_18-45-05.png
Screenshot_from_2024-04-19_18-45-05.png
Attachment 16: Screenshot_from_2024-04-19_22-47-43.png
Screenshot_from_2024-04-19_22-47-43.png
Attachment 17: Screenshot_from_2024-04-19_22-48-52.png
Screenshot_from_2024-04-19_22-48-52.png
Attachment 18: Screenshot_from_2024-04-19_22-49-31.png
Screenshot_from_2024-04-19_22-49-31.png
Attachment 19: Screenshot_from_2024-04-19_22-50-15.png
Screenshot_from_2024-04-19_22-50-15.png
Attachment 20: Screenshot_from_2024-04-19_22-51-23.png
Screenshot_from_2024-04-19_22-51-23.png
Attachment 21: Screenshot_from_2024-04-19_22-52-39.png
Screenshot_from_2024-04-19_22-52-39.png
Attachment 22: Screenshot_from_2024-04-19_22-52-14.png
Screenshot_from_2024-04-19_22-52-14.png
Attachment 23: Screenshot_from_2024-04-19_22-51-55.png
Screenshot_from_2024-04-19_22-51-55.png
  579   Fri Apr 19 02:35:14 2024 TDFriday 19 April
03.30


09.10



09.25 data file S100_alpha/R11
   
      BNC PB-5 pulser data

      amplitude 10.0V
      attenuation x10
      tau_d = 1ms
      polarity - ( initially + )
      frequency 25Hz

      ASIC settings2024Mar27-11.25.32
       slow comparator 0x64 all FEE64s


      aida02 pulser peak width 129 ch FWHM

      polsrity +

      aida09 pulser peak width 57 ch FWHM


09.40 switch to no storage

      p+n FEE64 slow comparators 0x64 -> 0xa
      n+n FEE64 slow comparators 0x64 -> 0xf

      radioactive source 152Eu 370kBq installed on the bottom side of the snout, centred tranversely, located beneath bplas-DSSSD-DSSSD-bplas position.

      ADC data item stats - attachment 35
      9/16 < 20k, max 108k

      per FEE64 Rate spectra - attachment 36

10.44 152Eu source removed

      ADC data item stats - attachment 37
      9/16 < 20k, max 102k

      per FEE64 Rate spectra - attachment 38
Attachment 1: Screenshot_from_2024-04-19_03-31-31.png
Screenshot_from_2024-04-19_03-31-31.png
Attachment 2: Screenshot_from_2024-04-19_03-30-58.png
Screenshot_from_2024-04-19_03-30-58.png
Attachment 3: Screenshot_from_2024-04-19_03-30-45.png
Screenshot_from_2024-04-19_03-30-45.png
Attachment 4: Screenshot_from_2024-04-19_03-27-45.png
Screenshot_from_2024-04-19_03-27-45.png
Attachment 5: Screenshot_from_2024-04-19_03-26-26.png
Screenshot_from_2024-04-19_03-26-26.png
Attachment 6: Screenshot_from_2024-04-19_03-25-53.png
Screenshot_from_2024-04-19_03-25-53.png
Attachment 7: Screenshot_from_2024-04-19_03-24-58.png
Screenshot_from_2024-04-19_03-24-58.png
Attachment 8: Screenshot_from_2024-04-19_09-08-46.png
Screenshot_from_2024-04-19_09-08-46.png
Attachment 9: Screenshot_from_2024-04-19_09-09-19.png
Screenshot_from_2024-04-19_09-09-19.png
Attachment 10: Screenshot_from_2024-04-19_09-10-19.png
Screenshot_from_2024-04-19_09-10-19.png
Attachment 11: Screenshot_from_2024-04-19_09-10-10.png
Screenshot_from_2024-04-19_09-10-10.png
Attachment 12: Screenshot_from_2024-04-19_09-10-03.png
Screenshot_from_2024-04-19_09-10-03.png
Attachment 13: Screenshot_from_2024-04-19_09-10-51.png
Screenshot_from_2024-04-19_09-10-51.png
Attachment 14: Screenshot_from_2024-04-19_09-11-08.png
Screenshot_from_2024-04-19_09-11-08.png
Attachment 15: Screenshot_from_2024-04-19_09-10-59.png
Screenshot_from_2024-04-19_09-10-59.png
Attachment 16: Screenshot_from_2024-04-19_09-11-20.png
Screenshot_from_2024-04-19_09-11-20.png
Attachment 17: Screenshot_from_2024-04-19_09-11-46.png
Screenshot_from_2024-04-19_09-11-46.png
Attachment 18: Screenshot_from_2024-04-19_09-11-31.png
Screenshot_from_2024-04-19_09-11-31.png
Attachment 19: Screenshot_from_2024-04-19_09-17-48.png
Screenshot_from_2024-04-19_09-17-48.png
Attachment 20: Screenshot_from_2024-04-19_09-17-22.png
Screenshot_from_2024-04-19_09-17-22.png
Attachment 21: Screenshot_from_2024-04-19_09-14-14.png
Screenshot_from_2024-04-19_09-14-14.png
Attachment 22: Screenshot_from_2024-04-19_09-13-23.png
Screenshot_from_2024-04-19_09-13-23.png
Attachment 23: Screenshot_from_2024-04-19_09-12-08.png
Screenshot_from_2024-04-19_09-12-08.png
Attachment 24: Screenshot_from_2024-04-19_09-12-01.png
Screenshot_from_2024-04-19_09-12-01.png
Attachment 25: Screenshot_from_2024-04-19_09-11-54.png
Screenshot_from_2024-04-19_09-11-54.png
Attachment 26: Screenshot_from_2024-04-19_09-18-40.png
Screenshot_from_2024-04-19_09-18-40.png
Attachment 27: Screenshot_from_2024-04-19_09-18-22.png
Screenshot_from_2024-04-19_09-18-22.png
Attachment 28: Screenshot_from_2024-04-19_09-18-01.png
Screenshot_from_2024-04-19_09-18-01.png
Attachment 29: Screenshot_from_2024-04-19_09-29-33.png
Screenshot_from_2024-04-19_09-29-33.png
Attachment 30: Screenshot_from_2024-04-19_09-27-40.png
Screenshot_from_2024-04-19_09-27-40.png
Attachment 31: Screenshot_from_2024-04-19_09-42-01.png
Screenshot_from_2024-04-19_09-42-01.png
Attachment 32: Screenshot_from_2024-04-19_09-41-31.png
Screenshot_from_2024-04-19_09-41-31.png
Attachment 33: Screenshot_from_2024-04-19_09-40-02.png
Screenshot_from_2024-04-19_09-40-02.png
Attachment 34: Screenshot_from_2024-04-19_09-37-03.png
Screenshot_from_2024-04-19_09-37-03.png
Attachment 35: Screenshot_from_2024-04-19_09-46-35.png
Screenshot_from_2024-04-19_09-46-35.png
Attachment 36: Screenshot_from_2024-04-19_09-47-08.png
Screenshot_from_2024-04-19_09-47-08.png
Attachment 37: Screenshot_from_2024-04-19_10-44-19.png
Screenshot_from_2024-04-19_10-44-19.png
Attachment 38: Screenshot_from_2024-04-19_10-43-58.png
Screenshot_from_2024-04-19_10-43-58.png
  578   Thu Apr 18 13:59:55 2024 TD, JBWR timestamp OK, readout time not OK
WR timestamps appear to be OK but readout times fort aida03, aida04 and aida09 appear to be out of sequence. Attachment 1-3.

Reseated HDMI cable of aida09 and performed complete powercycle. Check the MACB HDMI port (appears to be loose) - still no change aida09 not collecting WR timestamps. Attachment 4-5.
Attachment 1: Screenshot_from_2024-04-18_14-59-38.png
Screenshot_from_2024-04-18_14-59-38.png
Attachment 2: Screenshot_from_2024-04-18_15-01-15.png
Screenshot_from_2024-04-18_15-01-15.png
Attachment 3: Screenshot_from_2024-04-18_15-01-10.png
Screenshot_from_2024-04-18_15-01-10.png
Attachment 4: Screenshot_from_2024-04-18_16-13-55.png
Screenshot_from_2024-04-18_16-13-55.png
Attachment 5: Screenshot_from_2024-04-18_16-13-48.png
Screenshot_from_2024-04-18_16-13-48.png
  577   Thu Apr 18 08:57:48 2024 JB, TDThursday 18 April
09.52 DAQ continues file S100_alpha/R5
       note merger not functioning correctly due to aida03, aida04 & aida09 out of WR sequence 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      WR timestamps - attachment 3
       aida03, aida04 & aida09 out of sequence

      All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 4-6

      ADC data item stats - attachment 7

      per FEE64 Stat spectra - attachment 8

      per FEE64 1.8.L spectra - attachments 9-10

      per FEE64 1.8.W spectra - 20us FSR - attachments 11-12

      Merger, TapeServer & Link Data Rates - attachments 13-15

      Grafana - attachment 16
       DSSSD bias/leakage current trends OK


19.15 alpha background file S100_alpha/R10

      DSSSD bias & leakage current OK - attachment 17

      FEE64 temps OK - attachment 18
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* ADC calibratio/FPGA/WR decoder status - attachments 19-21

      WR timestamps OK - attachment 22

      ADC data item stats - attachment 23

      Merger, TapeServer & Meger stats - attachments 24-26
       data rate c. 2.8Mb/s

      per FEE64 1.8.W spectra - 20us FSR - attachments 27-28

      correlation scaler stats & data link stats - attachments 29-30
       data rate dominated by MACB scaler inputs aida03 & aida04 ( c. 200kHz each )


22.16 DAQ continues Ok file S100_alpha/R10_35

      Merger, TapeServer & Meger stats - attachments 31-33
       data rate c. 2.8Mb/s

      ADC data item stats - attachment 34

      FEE64 temps OK - attachment 35
      *except* aida02 ASIC temp which is known to be u/s

      DSSSD bias & leakage current OK - attachment 36
Attachment 1: Screenshot_from_2024-04-18_09-52-14.png
Screenshot_from_2024-04-18_09-52-14.png
Attachment 2: Screenshot_from_2024-04-18_09-52-28.png
Screenshot_from_2024-04-18_09-52-28.png
Attachment 3: Screenshot_from_2024-04-18_09-53-30.png
Screenshot_from_2024-04-18_09-53-30.png
Attachment 4: Screenshot_from_2024-04-18_09-52-57.png
Screenshot_from_2024-04-18_09-52-57.png
Attachment 5: Screenshot_from_2024-04-18_09-52-45.png
Screenshot_from_2024-04-18_09-52-45.png
Attachment 6: Screenshot_from_2024-04-18_09-52-40.png
Screenshot_from_2024-04-18_09-52-40.png
Attachment 7: Screenshot_from_2024-04-18_09-53-48.png
Screenshot_from_2024-04-18_09-53-48.png
Attachment 8: Screenshot_from_2024-04-18_09-56-18.png
Screenshot_from_2024-04-18_09-56-18.png
Attachment 9: Screenshot_from_2024-04-18_09-55-07.png
Screenshot_from_2024-04-18_09-55-07.png
Attachment 10: Screenshot_from_2024-04-18_09-54-11.png
Screenshot_from_2024-04-18_09-54-11.png
Attachment 11: Screenshot_from_2024-04-18_09-57-30.png
Screenshot_from_2024-04-18_09-57-30.png
Attachment 12: Screenshot_from_2024-04-18_09-56-48.png
Screenshot_from_2024-04-18_09-56-48.png
Attachment 13: Screenshot_from_2024-04-18_09-59-16.png
Screenshot_from_2024-04-18_09-59-16.png
Attachment 14: Screenshot_from_2024-04-18_09-59-26.png
Screenshot_from_2024-04-18_09-59-26.png
Attachment 15: Screenshot_from_2024-04-18_09-59-38.png
Screenshot_from_2024-04-18_09-59-38.png
Attachment 16: Screenshot_from_2024-04-18_10-10-43.png
Screenshot_from_2024-04-18_10-10-43.png
Attachment 17: Screenshot_from_2024-04-18_19-12-32.png
Screenshot_from_2024-04-18_19-12-32.png
Attachment 18: Screenshot_from_2024-04-18_19-12-47.png
Screenshot_from_2024-04-18_19-12-47.png
Attachment 19: Screenshot_from_2024-04-18_19-13-11.png
Screenshot_from_2024-04-18_19-13-11.png
Attachment 20: Screenshot_from_2024-04-18_19-13-04.png
Screenshot_from_2024-04-18_19-13-04.png
Attachment 21: Screenshot_from_2024-04-18_19-12-57.png
Screenshot_from_2024-04-18_19-12-57.png
Attachment 22: Screenshot_from_2024-04-18_19-13-51.png
Screenshot_from_2024-04-18_19-13-51.png
Attachment 23: Screenshot_from_2024-04-18_19-14-02.png
Screenshot_from_2024-04-18_19-14-02.png
Attachment 24: Screenshot_from_2024-04-18_19-15-10.png
Screenshot_from_2024-04-18_19-15-10.png
Attachment 25: Screenshot_from_2024-04-18_19-14-32.png
Screenshot_from_2024-04-18_19-14-32.png
Attachment 26: Screenshot_from_2024-04-18_19-14-12.png
Screenshot_from_2024-04-18_19-14-12.png
Attachment 27: Screenshot_from_2024-04-18_19-12-14.png
Screenshot_from_2024-04-18_19-12-14.png
Attachment 28: Screenshot_from_2024-04-18_19-11-30.png
Screenshot_from_2024-04-18_19-11-30.png
Attachment 29: Screenshot_from_2024-04-18_19-31-12.png
Screenshot_from_2024-04-18_19-31-12.png
Attachment 30: Screenshot_from_2024-04-18_19-41-59.png
Screenshot_from_2024-04-18_19-41-59.png
Attachment 31: Screenshot_from_2024-04-18_22-16-42.png
Screenshot_from_2024-04-18_22-16-42.png
Attachment 32: Screenshot_from_2024-04-18_22-16-04.png
Screenshot_from_2024-04-18_22-16-04.png
Attachment 33: Screenshot_from_2024-04-18_22-15-41.png
Screenshot_from_2024-04-18_22-15-41.png
Attachment 34: Screenshot_from_2024-04-18_22-12-53.png
Screenshot_from_2024-04-18_22-12-53.png
Attachment 35: Screenshot_from_2024-04-18_22-12-13.png
Screenshot_from_2024-04-18_22-12-13.png
Attachment 36: Screenshot_from_2024-04-18_22-10-53.png
Screenshot_from_2024-04-18_22-10-53.png
  576   Tue Apr 16 23:37:58 2024 TDWednesday 17 April
00.40 Observed zero data fron aida03, aida04, aida09
      WR timestamps offset from other FEE64s

      rebooted aida03, aida04, aida09
      no change
      see attachments 14-17



18.25 rebooted aida03, aida04, aida09, aida15

      system wide checks OK *except* aida15 ADC not calibrated, FPGA decoder errors aida02, aida03
      ADC data item stats - attachment 28
      WR timestamps OK - attachment 29

      data file S100_alpha/R5 - alpha background

      Merger and TapeServer - attachments 30-31
       c. 1Mb/s to disk

21.05 FEE64 temps OK - attachment 44
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida15 ASDC not calibrated, WR timestam/FPGA errors - attachments 41-43

      WR timestamps- attachment 40
       aida03, aida04 and aida09 differ from other timestamps again

      Merger and TapeServer - attachments 38-39
       c. 0Mb/s to disk due to WR timestamp issues

      per 1.8.W spectra - 20us FSR - attachments 36-37

      per FEE64 Rate spectra - attachments 34-35

      per FEE64 1.8.L spectra - attachments 32-33
Attachment 1: Screenshot_from_2024-04-17_00-35-53.png
Screenshot_from_2024-04-17_00-35-53.png
Attachment 2: Screenshot_from_2024-04-17_00-34-46.png
Screenshot_from_2024-04-17_00-34-46.png
Attachment 3: Screenshot_from_2024-04-17_00-33-46.png
Screenshot_from_2024-04-17_00-33-46.png
Attachment 4: Screenshot_from_2024-04-17_00-32-53.png
Screenshot_from_2024-04-17_00-32-53.png
Attachment 5: Screenshot_from_2024-04-17_00-31-43.png
Screenshot_from_2024-04-17_00-31-43.png
Attachment 6: Screenshot_from_2024-04-17_00-30-24.png
Screenshot_from_2024-04-17_00-30-24.png
Attachment 7: Screenshot_from_2024-04-17_00-29-34.png
Screenshot_from_2024-04-17_00-29-34.png
Attachment 8: Screenshot_from_2024-04-17_00-28-43.png
Screenshot_from_2024-04-17_00-28-43.png
Attachment 9: Screenshot_from_2024-04-17_00-27-54.png
Screenshot_from_2024-04-17_00-27-54.png
Attachment 10: Screenshot_from_2024-04-17_00-27-31.png
Screenshot_from_2024-04-17_00-27-31.png
Attachment 11: Screenshot_from_2024-04-17_00-27-05.png
Screenshot_from_2024-04-17_00-27-05.png
Attachment 12: Screenshot_from_2024-04-17_00-26-32.png
Screenshot_from_2024-04-17_00-26-32.png
Attachment 13: Screenshot_from_2024-04-17_00-25-51.png
Screenshot_from_2024-04-17_00-25-51.png
Attachment 14: Screenshot_from_2024-04-17_01-05-47.png
Screenshot_from_2024-04-17_01-05-47.png
Attachment 15: Screenshot_from_2024-04-17_01-05-09.png
Screenshot_from_2024-04-17_01-05-09.png
Attachment 16: Screenshot_from_2024-04-17_01-01-20.png
Screenshot_from_2024-04-17_01-01-20.png
Attachment 17: Screenshot_from_2024-04-17_01-00-33.png
Screenshot_from_2024-04-17_01-00-33.png
Attachment 18: Screenshot_from_2024-04-17_18-00-08.png
Screenshot_from_2024-04-17_18-00-08.png
Attachment 19: Screenshot_from_2024-04-17_17-59-42.png
Screenshot_from_2024-04-17_17-59-42.png
Attachment 20: Screenshot_from_2024-04-17_17-58-30.png
Screenshot_from_2024-04-17_17-58-30.png
Attachment 21: Screenshot_from_2024-04-17_17-57-46.png
Screenshot_from_2024-04-17_17-57-46.png
Attachment 22: Screenshot_from_2024-04-17_17-57-12.png
Screenshot_from_2024-04-17_17-57-12.png
Attachment 23: Screenshot_from_2024-04-17_17-56-27.png
Screenshot_from_2024-04-17_17-56-27.png
Attachment 24: Screenshot_from_2024-04-17_17-55-54.png
Screenshot_from_2024-04-17_17-55-54.png
Attachment 25: Screenshot_from_2024-04-17_17-55-33.png
Screenshot_from_2024-04-17_17-55-33.png
Attachment 26: Screenshot_from_2024-04-17_17-54-35.png
Screenshot_from_2024-04-17_17-54-35.png
Attachment 27: Screenshot_from_2024-04-17_17-54-04.png
Screenshot_from_2024-04-17_17-54-04.png
Attachment 28: Screenshot_from_2024-04-17_18-23-26.png
Screenshot_from_2024-04-17_18-23-26.png
Attachment 29: Screenshot_from_2024-04-17_18-22-42.png
Screenshot_from_2024-04-17_18-22-42.png
Attachment 30: Screenshot_from_2024-04-17_18-25-37.png
Screenshot_from_2024-04-17_18-25-37.png
Attachment 31: Screenshot_from_2024-04-17_18-25-08.png
Screenshot_from_2024-04-17_18-25-08.png
Attachment 32: Screenshot_from_2024-04-17_21-06-56.png
Screenshot_from_2024-04-17_21-06-56.png
Attachment 33: Screenshot_from_2024-04-17_21-05-45.png
Screenshot_from_2024-04-17_21-05-45.png
Attachment 34: Screenshot_from_2024-04-17_21-04-52.png
Screenshot_from_2024-04-17_21-04-52.png
Attachment 35: Screenshot_from_2024-04-17_21-04-17.png
Screenshot_from_2024-04-17_21-04-17.png
Attachment 36: Screenshot_from_2024-04-17_21-03-15.png
Screenshot_from_2024-04-17_21-03-15.png
Attachment 37: Screenshot_from_2024-04-17_21-02-29.png
Screenshot_from_2024-04-17_21-02-29.png
Attachment 38: Screenshot_from_2024-04-17_21-01-57.png
Screenshot_from_2024-04-17_21-01-57.png
Attachment 39: Screenshot_from_2024-04-17_21-01-32.png
Screenshot_from_2024-04-17_21-01-32.png
Attachment 40: Screenshot_from_2024-04-17_21-00-34.png
Screenshot_from_2024-04-17_21-00-34.png
Attachment 41: Screenshot_from_2024-04-17_20-59-40.png
Screenshot_from_2024-04-17_20-59-40.png
Attachment 42: Screenshot_from_2024-04-17_20-59-16.png
Screenshot_from_2024-04-17_20-59-16.png
Attachment 43: Screenshot_from_2024-04-17_20-58-54.png
Screenshot_from_2024-04-17_20-58-54.png
Attachment 44: Screenshot_from_2024-04-17_20-58-02.png
Screenshot_from_2024-04-17_20-58-02.png
  575   Tue Apr 16 10:49:37 2024 TD, JB, CCTuesday 16 April
11.45 DSSSD bias & leakage current OK - attachments 9-10

      FEE64 temps OK - attachment 8
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida04 WR decoder/FPGA status errors - attachments 6-7
       HDMI cabling disturbed during platform move?

      WR timestamps OK - attachment 5

      ADC data item stats - attachment 4
        10/16 < 20k, max 115k

      per FEE64 Rate spectra - attachments 3

      per 1.8.W spectra - 20us FSR - attachments 1-2

11.58 BB7 preamps unplugged. Noise conditions OK. Did not change considerably.
      11/16 < 20k (aida09 has 1 hot channel), max 114k Attachments 11-12.

12:34 Detector powered down to bring to safe state to check HDMI cables of aida02 and aida03.

15:06 Reseated aida02 and aida03. No more FPGA timestamp error. See attachment 13-14.

      Rates and histograms look good for pulser walk through. Test + and test - connected. Currently, Attachment 16-17.
      Temp OK Attachment 15. Bias OK Attachment 18.

      Data links aida02 and aida08 missing (?)

16:30 All slow comparator thresholds set to 0x64 for alpha background run during platform freeze.

      Initial rates and histograms attachment 19 - 20. aida04 noisy in ASIC 1.

      Temps OK. Attachment 21.
      MBS disc and tape setup in atttachment 22-23.
Attachment 1: Screenshot_from_2024-04-16_11-49-17.png
Screenshot_from_2024-04-16_11-49-17.png
Attachment 2: Screenshot_from_2024-04-16_11-48-28.png
Screenshot_from_2024-04-16_11-48-28.png
Attachment 3: Screenshot_from_2024-04-16_11-47-40.png
Screenshot_from_2024-04-16_11-47-40.png
Attachment 4: Screenshot_from_2024-04-16_11-46-48.png
Screenshot_from_2024-04-16_11-46-48.png
Attachment 5: Screenshot_from_2024-04-16_11-46-01.png
Screenshot_from_2024-04-16_11-46-01.png
Attachment 6: Screenshot_from_2024-04-16_11-45-15.png
Screenshot_from_2024-04-16_11-45-15.png
Attachment 7: Screenshot_from_2024-04-16_11-44-37.png
Screenshot_from_2024-04-16_11-44-37.png
Attachment 8: Screenshot_from_2024-04-16_11-44-06.png
Screenshot_from_2024-04-16_11-44-06.png
Attachment 9: Screenshot_from_2024-04-16_11-51-17.png
Screenshot_from_2024-04-16_11-51-17.png
Attachment 10: Screenshot_from_2024-04-16_11-50-22.png
Screenshot_from_2024-04-16_11-50-22.png
Attachment 11: Screenshot_from_2024-04-16_11-58-26.png
Screenshot_from_2024-04-16_11-58-26.png
Attachment 12: Screenshot_from_2024-04-16_11-58-08.png
Screenshot_from_2024-04-16_11-58-08.png
Attachment 13: Screenshot_from_2024-04-16_15-04-13.png
Screenshot_from_2024-04-16_15-04-13.png
Attachment 14: Screenshot_from_2024-04-16_15-04-07.png
Screenshot_from_2024-04-16_15-04-07.png
Attachment 15: Screenshot_from_2024-04-16_15-06-31.png
Screenshot_from_2024-04-16_15-06-31.png
Attachment 16: Screenshot_from_2024-04-16_15-06-12.png
Screenshot_from_2024-04-16_15-06-12.png
Attachment 17: Screenshot_from_2024-04-16_15-05-33.png
Screenshot_from_2024-04-16_15-05-33.png
Attachment 18: Screenshot_from_2024-04-16_15-08-04.png
Screenshot_from_2024-04-16_15-08-04.png
Attachment 19: Screenshot_from_2024-04-16_15-10-46.png
Screenshot_from_2024-04-16_15-10-46.png
Attachment 20: Screenshot_from_2024-04-16_17-17-35.png
Screenshot_from_2024-04-16_17-17-35.png
Attachment 21: Screenshot_from_2024-04-16_17-17-32.png
Screenshot_from_2024-04-16_17-17-32.png
Attachment 22: Screenshot_from_2024-04-16_17-14-40.png
Screenshot_from_2024-04-16_17-14-40.png
Attachment 23: Screenshot_from_2024-04-16_17-17-24.png
Screenshot_from_2024-04-16_17-17-24.png
Attachment 24: Screenshot_from_2024-04-16_17-17-11.png
Screenshot_from_2024-04-16_17-17-11.png
  574   Mon Apr 15 09:23:43 2024 CC, JB, HMA, TDNoise in AIDA when connecting BB7

9:40 started up AIDA. TEMP OK. Noise condition same as Saturday. Attachments 1-3.

9:56 BB7 premaps connected but not powered. Attahcment 4.

10:20 BB7 Preamps powered. Attachment 5.

Noise increased marginally, but still 10/16 <20kHz, max 130k.

 

 

15.15  per FEE64 Rate spectra - attachment 6

             ADC data item stats - attachment 7

           increased noise observed - 6/16 < 20k, max 300k 

 

17.10  per FEE64 1.8.W spectra - 20us FSR - attachments 8-9

            per FEE64 Rate spectra - attachment 10

             ADC data item stats - attachment 11

           increased noise observed - 7/16 < 20k, max 180k 

20.30  per FEE64 Rate spectra - attachment 12

             ADC data item stats - attachment 13

           status quo ante - 10/16 < 20k, max 130k 

 

          FEE64 power OFF

          DSSSD bias OFF

 

 

Attachment 1: Screenshot_from_2024-04-15_09-55-15.png
Screenshot_from_2024-04-15_09-55-15.png
Attachment 2: Screenshot_from_2024-04-15_09-54-58.png
Screenshot_from_2024-04-15_09-54-58.png
Attachment 3: Screenshot_from_2024-04-15_09-54-17.png
Screenshot_from_2024-04-15_09-54-17.png
Attachment 4: Screenshot_from_2024-04-15_10-13-20.png
Screenshot_from_2024-04-15_10-13-20.png
Attachment 5: Screenshot_from_2024-04-15_10-24-39.png
Screenshot_from_2024-04-15_10-24-39.png
Attachment 6: Screenshot_from_2024-04-15_15-24-34.png
Screenshot_from_2024-04-15_15-24-34.png
Attachment 7: Screenshot_from_2024-04-15_15-23-41.png
Screenshot_from_2024-04-15_15-23-41.png
Attachment 8: Screenshot_from_2024-04-15_17-16-19.png
Screenshot_from_2024-04-15_17-16-19.png
Attachment 9: Screenshot_from_2024-04-15_17-15-27.png
Screenshot_from_2024-04-15_17-15-27.png
Attachment 10: Screenshot_from_2024-04-15_17-14-32.png
Screenshot_from_2024-04-15_17-14-32.png
Attachment 11: Screenshot_from_2024-04-15_17-13-37.png
Screenshot_from_2024-04-15_17-13-37.png
Attachment 12: Screenshot_from_2024-04-15_20-29-18.png
Screenshot_from_2024-04-15_20-29-18.png
Attachment 13: Screenshot_from_2024-04-15_20-28-48.png
Screenshot_from_2024-04-15_20-28-48.png
  573   Sat Apr 13 14:04:44 2024 JB13 April Noise checks

15:04 bPlast was left powered over night. Powering up to check noise conditions - also to check shifting of Germanium baseline (no change observed).

          TEMP OK.

          Noise condition the same as yesterday when platform was moved in. Noise in aida01 and aida09 due to single channel failures (?). Attachments 1-3.

          9/16 <20kHz, max 176k

15:30 Changed drainwires ground from 4V (Ch3) -> 29.5V (Ch4) out on R&SMP4040 PSU. Attachments 4-5.

          No change in noise observed.

          9/16 <20kHz, max 187k

15:49 Connected PN 300 ground to R&SMP4040 4V (Ch3) output. Noise decreased in many channels aida09 does not have a noisy channel anymore (??). Attachments 6-7.

          10/16 <20kHz, max 158k.

 

Current bPlast grounding scheme in attachments 8-9.

 

Attachment 1: Screenshot_from_2024-04-13_15-21-34.png
Screenshot_from_2024-04-13_15-21-34.png
Attachment 2: Screenshot_from_2024-04-13_15-20-28.png
Screenshot_from_2024-04-13_15-20-28.png
Attachment 3: Screenshot_from_2024-04-13_15-15-14.png
Screenshot_from_2024-04-13_15-15-14.png
Attachment 4: Screenshot_from_2024-04-13_15-40-59.png
Screenshot_from_2024-04-13_15-40-59.png
Attachment 5: Screenshot_from_2024-04-13_15-40-56.png
Screenshot_from_2024-04-13_15-40-56.png
Attachment 6: Screenshot_from_2024-04-13_15-48-53.png
Screenshot_from_2024-04-13_15-48-53.png
Attachment 7: Screenshot_from_2024-04-13_15-48-42.png
Screenshot_from_2024-04-13_15-48-42.png
Attachment 8: 20240413_155823.jpg
20240413_155823.jpg
Attachment 9: 20240413_160214.jpg
20240413_160214.jpg
  572   Thu Apr 11 22:04:50 2024 JB12.04.2024 AIDA-bPlast noise optimisation

15:00 Platform in, biasing detector. Temp OK Attachment 1. test - Pulser OFF.

           Rates somewhat worse than last night - probable contact on grounding of snout. Attachment 3.

           Histograms okay, some FEE64s now have hot channels. Attachment 2.

           9/16 aidas < 20kHz - max rate 162k

 

18:06 - powering down detector for the weekend. Overall system is fine, some channels (hot channels) definitely picked up noise, but condition is overall stable over three hours. See atachments 4-5.

 

TO-DO (kicking the can down the road):

- Some work to do on noise, but we might have to accept the situation as it currently is.

- Implement bPlast trigger scheme. Set bPlast thresholds, get bPlast current draw undercontrol - observe AIDA noise.

- Try different grounding configuration:

    - Grounding drain wires to frame. Observe AIDA conditions.

    - Reconnect 4V out to R&SRMP4040 common ground with PN 300 PSU as well. Observe AIDA conditions.

 

Attachment 1: Screenshot_from_2024-04-12_15-11-18.png
Screenshot_from_2024-04-12_15-11-18.png
Attachment 2: Screenshot_from_2024-04-12_15-22-54.png
Screenshot_from_2024-04-12_15-22-54.png
Attachment 3: Screenshot_from_2024-04-12_15-22-26.png
Screenshot_from_2024-04-12_15-22-26.png
Attachment 4: Screenshot_from_2024-04-12_18-07-35.png
Screenshot_from_2024-04-12_18-07-35.png
Attachment 5: Screenshot_from_2024-04-12_18-07-27.png
Screenshot_from_2024-04-12_18-07-27.png
  571   Thu Apr 11 08:27:21 2024 JB, TD11 April noise tests
TO-DO for 11.04.2024

- Try bringing bPlast drain wire ground back to the PSU ground for PN 300 and R&SRMP4040.

- Recheck the downstream detector bias and ground scheme.

-

 

9:27 Restarted AIDA and hit go at 9:54. Situation the same as it was in the evening of 10.04.2024. 10 out of 16 FEE64s in good condition. Waveforms of aida16,

        TEMP OK - HV OK. See attachments 1-6.

 

10:20 We grounded the output of the R&SRMP4040 to to the drain wire ground of the bPlast ribbon cables and the output of the PN 300 PSU that is powering the booster boards. Results given by attachments 7 -11. We only saw a marginal improvement in the overall noise condition.

10:32 We connected the ground of the frame to the drain wire ground, that is also mutually connected to the ground of the R&SRMP4040 output ground and PN 300 output ground. Results given by attachments ???. We did not see any improvement in the overal noise condition.

 

11.00 DSSSD HV OFF

          FEE64 power OFF

          Check seating of all adaptor PCBs and drain/ground wires secured - generally OK

 

          Restart

          1.8.W spectra - 20us FSR - attachments 15-16

          per FEE64 Rate spectra - attachment 17

           10/16 < 20k, max 110k

          ADC data item stats - attachment 18

          Incremental improvement.

 

11.45 DSSSD HV OFF

          FEE64 power OFF

          test - daisy chain removed

 

          Restart

          1.8.W spectra - 20us FSR - attachments 20-21

          per FEE64 Rate spectra - attachment 19

           10/16 < 20k, max 210k

          ADC data item stats - attachment 22

          overall somewhat worse

 

12.20 DSSSD HV OFF

          FEE64 power OFF

          Re-install test - daisy chain, Tighten aida04 DSSSD ribbon cable drain wire.

 

          Restart

          1.8.W spectra - 20us FSR - attachments 23-24

          per FEE64 Rate spectra - attachment 25

          ADC data item stats - attachment 26

           10/16 < 20k, max 110k

          Status quo ante

 

         aida04 & aida08 1*W & 2*W spectra - 200us FSR - attachments 27-31

           large transients observed for aida04 1*W - cable or ASIC fault?

 

         aida16 **W spectra - 200us FSR - attachments 32-35

           all channels apopear to be working but mix of high/low noise channels for asics #3-4, asics #1-2 all appear high noise

 

       

 

 

        

14.00 DSSSD HV OFF

          FEE64 power OFF

          Disconnect ribbon cables from aida04 adaptor PCB

 

          Restart

          n+n FEE64 1.8.W spectra - 20us FSR - attachments 37

          per FEE64 Rate spectra - attachment 36

          ADC data item stats - attachment 38

          Implies origin of large transients observed in aida04 asic#1 is downstream of the FEE64 adaptor PCB, i.e. ribbon cable or DSSSD.

 

 

14.40 DSSSD HV OFF

          FEE64 power OFF

          re-connect ribbon cables from aida04 adaptor PCB

          1x pin J2 slightly bent - straightended with screwdriver - FFSD connector insertion OK

 

          Restart

          n+n FEE64 1.8.W spectra - 20us FSR - attachments 40-41

          per FEE64 Rate spectra - attachment 42-43

          ADC data item stats - attachment 39

            10x < 20k, max 120k

 

 

           aida11 asic#4 1.8.W spectra - 200us FSR - attachment

             no large transients observed

 

           DSSSD #1 & #2 bias from -120V to -100V

            ADC data item stats - attachment 40

            9x < 20k, max 220k (aida08)

            per FEE64 Rate spectra - attachment 41

 

17.15 bPlas ON -

        Current bPlast ground configuration:

         bPlas current ground configuration - drainwires of all ribbon cables excluding short side (cont. with snout) are grounded back to the bPlast R&SRMP4040 PSU, on the 29 V output. 4 V PSU to booster boards are floating, output of PN 300 is grounded to the PSU. The mesytec PSU, that also powers bPlast SiPMs at

         29V is not grounded to anything but the frame. Snout is currently light tight, I suspect internal radiation from FATIMA.

         It is noted that bPlast current draw fluctuated significantly (+/- 300 mV) as a result of thresholds set to the detector. I tried to set the thresholds HIGH to stabilise the bPlast detector as the power draw is fluctuating greatly.         

          AIDA noise very good

            ADC data item stats - attachment 42

            12x < 20k, max 83k (aida04)

            per FEE64 Rate spectra - attachment 43

           

 

 

To Do list

- separate 29V (low current) and 4V (high current) return paths/ground refs

- The mesytec PSU that powers the bPlast detector  is running to frame, it might be an idea to ground ribbon c

 

Attachment 1: Screenshot_from_2024-04-11_09-56-51.png
Screenshot_from_2024-04-11_09-56-51.png
Attachment 2: Screenshot_from_2024-04-11_09-55-50.png
Screenshot_from_2024-04-11_09-55-50.png
Attachment 3: Screenshot_from_2024-04-11_09-55-21.png
Screenshot_from_2024-04-11_09-55-21.png
Attachment 4: Screenshot_from_2024-04-11_09-55-10.png
Screenshot_from_2024-04-11_09-55-10.png
Attachment 5: Screenshot_from_2024-04-11_09-54-19.png
Screenshot_from_2024-04-11_09-54-19.png
Attachment 6: Screenshot_from_2024-04-11_09-53-48.png
Screenshot_from_2024-04-11_09-53-48.png
Attachment 7: Screenshot_from_2024-04-11_10-21-37.png
Screenshot_from_2024-04-11_10-21-37.png
Attachment 8: Screenshot_from_2024-04-11_10-21-08.png
Screenshot_from_2024-04-11_10-21-08.png
Attachment 9: Screenshot_from_2024-04-11_10-18-40.png
Screenshot_from_2024-04-11_10-18-40.png
Attachment 10: Screenshot_from_2024-04-11_10-18-38.png
Screenshot_from_2024-04-11_10-18-38.png
Attachment 11: Screenshot_from_2024-04-11_10-38-51.png
Screenshot_from_2024-04-11_10-38-51.png
Attachment 12: Screenshot_from_2024-04-11_10-37-16.png
Screenshot_from_2024-04-11_10-37-16.png
Attachment 13: Screenshot_from_2024-04-11_10-36-23.png
Screenshot_from_2024-04-11_10-36-23.png
Attachment 14: Screenshot_from_2024-04-11_10-36-13.png
Screenshot_from_2024-04-11_10-36-13.png
Attachment 15: Screenshot_from_2024-04-11_11-18-05.png
Screenshot_from_2024-04-11_11-18-05.png
Attachment 16: Screenshot_from_2024-04-11_11-17-16.png
Screenshot_from_2024-04-11_11-17-16.png
Attachment 17: Screenshot_from_2024-04-11_11-14-20.png
Screenshot_from_2024-04-11_11-14-20.png
Attachment 18: Screenshot_from_2024-04-11_11-14-08.png
Screenshot_from_2024-04-11_11-14-08.png
Attachment 19: Screenshot_from_2024-04-11_11-47-55.png
Screenshot_from_2024-04-11_11-47-55.png
Attachment 20: Screenshot_from_2024-04-11_11-46-52.png
Screenshot_from_2024-04-11_11-46-52.png
Attachment 21: Screenshot_from_2024-04-11_11-46-01.png
Screenshot_from_2024-04-11_11-46-01.png
Attachment 22: Screenshot_from_2024-04-11_11-45-32.png
Screenshot_from_2024-04-11_11-45-32.png
Attachment 23: Screenshot_from_2024-04-11_12-24-14.png
Screenshot_from_2024-04-11_12-24-14.png
Attachment 24: Screenshot_from_2024-04-11_12-23-39.png
Screenshot_from_2024-04-11_12-23-39.png
Attachment 25: Screenshot_from_2024-04-11_12-23-08.png
Screenshot_from_2024-04-11_12-23-08.png
Attachment 26: Screenshot_from_2024-04-11_12-22-27.png
Screenshot_from_2024-04-11_12-22-27.png
Attachment 27: Screenshot_from_2024-04-11_12-35-36.png
Screenshot_from_2024-04-11_12-35-36.png
Attachment 28: Screenshot_from_2024-04-11_12-35-07.png
Screenshot_from_2024-04-11_12-35-07.png
Attachment 29: Screenshot_from_2024-04-11_12-34-18.png
Screenshot_from_2024-04-11_12-34-18.png
Attachment 30: Screenshot_from_2024-04-11_12-33-13.png
Screenshot_from_2024-04-11_12-33-13.png
Attachment 31: Screenshot_from_2024-04-11_12-37-22.png
Screenshot_from_2024-04-11_12-37-22.png
Attachment 32: Screenshot_from_2024-04-11_12-44-59.png
Screenshot_from_2024-04-11_12-44-59.png
Attachment 33: Screenshot_from_2024-04-11_12-43-07.png
Screenshot_from_2024-04-11_12-43-07.png
Attachment 34: Screenshot_from_2024-04-11_12-42-19.png
Screenshot_from_2024-04-11_12-42-19.png
Attachment 35: Screenshot_from_2024-04-11_12-40-03.png
Screenshot_from_2024-04-11_12-40-03.png
Attachment 36: Screenshot_from_2024-04-11_14-04-59.png
Screenshot_from_2024-04-11_14-04-59.png
Attachment 37: Screenshot_from_2024-04-11_14-05-23.png
Screenshot_from_2024-04-11_14-05-23.png
Attachment 38: Screenshot_from_2024-04-11_14-19-10.png
Screenshot_from_2024-04-11_14-19-10.png
Attachment 39: Screenshot_from_2024-04-11_14-48-09.png
Screenshot_from_2024-04-11_14-48-09.png
Attachment 40: Screenshot_from_2024-04-11_14-48-29.png
Screenshot_from_2024-04-11_14-48-29.png
Attachment 41: Screenshot_from_2024-04-11_14-48-57.png
Screenshot_from_2024-04-11_14-48-57.png
Attachment 42: Screenshot_from_2024-04-11_14-49-37.png
Screenshot_from_2024-04-11_14-49-37.png
Attachment 43: Screenshot_from_2024-04-11_14-49-26.png
Screenshot_from_2024-04-11_14-49-26.png
Attachment 44: Screenshot_from_2024-04-11_15-11-42.png
Screenshot_from_2024-04-11_15-11-42.png
Attachment 45: Screenshot_from_2024-04-11_15-17-34.png
Screenshot_from_2024-04-11_15-17-34.png
Attachment 46: Screenshot_from_2024-04-11_15-17-10.png
Screenshot_from_2024-04-11_15-17-10.png
Attachment 47: Screenshot_from_2024-04-11_17-13-36.png
Screenshot_from_2024-04-11_17-13-36.png
Attachment 48: Screenshot_from_2024-04-11_17-13-40.png
Screenshot_from_2024-04-11_17-13-40.png
  570   Wed Apr 10 08:37:57 2024 JB, CC, TDWednesday 10 April
09.38 CC completed install of Bplast driver PCBs yesterday evening.
      All flat ribbon cables connected - all drain wires grounded
      PSU on but not enabled - return terminals grounded to PSU front panel ground
      SiPm bias off

      Water pressure and temperature OK
      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3

      WR timestamps OK - attachment 4

      ADC data item stats - attachments 5

      per FEE64 Rate spectra - attachments 6-7

      per 1.8.W spectra - 20us FSR - attachments 8-11

      ASIC settings 2024Mar27-11.25.32
       LEC slow comparator p+n FEE64s 0xa, n+n FEE64s 0xf

      BNC PB-5 pulser - attachment 12



10.45 CC returns
      bPlas ON

      ADC data item stats - attachments 13
       8x < 20k, max c. 310k

      per FEE64 Rate spectra - attachments 14

      per 1.8.W spectra - 20us FSR - attachments 15-16

      downstream DSSSD n+n and bottom left & right p+n FEE64s noisy


11.40 per p+n FEE64 1.8.L spectra - attachment 17
       aida09 pulser peak width 55 ch FWHM ~38keV FWHM - no change cf. before installation of bPas *except* aida16


12.00 Photos of snout, bPlas driver PCBs, cabling, grounding and PSUs courtesy JB - attachments 18-27

      N.B outputs of PN300 PSU at base of AIDA support stand are *not* ground ref'd - attachment 23


12.20 Slow comparator -> 0x64
      Pulser OFF
      All histograms zero'd

13.17 JB: returned from lunch. Current status of AIDA modules given by attachment 29.

15.55 While bPlast thresholds were being set the noise increased substantially.

16.51 Replaced mezzanine of aida14, reinstalled and biasing detector. Resulting for noise conditions in the detector given by attachments 30-33.

17.31 With a multimeter it was found that there is continuity between the snout and the frame.

18.50 We tried to disconnect the cables of the short side of the bPlast detector and all of the grounds. This seemed to show an open line OL on the multimeter and whence connecting the bPlast grounds back excluding the short side ribbon cable grounds, the multimeter still read OL. 
      We also tried to wedge paper between the short side ribbon cable of bPlast and the snout, but this did not work, the detector still reading continuity between the frame, booster board and snout. 
      The results after booting up the detector again are given by the attachments 34-36. 
      The noise condition is appreciably better than before with 12 out of the 16 FEE64 modules with sub 20 kHz rates.
      Disconnect the BB7 preamp. ground from the frame.

18.54 We powered up bPlast. The noise condition three FEEs got worse, aida01, aida11 and 06. The problem may be associated with downstream grounding. The results are given by the attachments 37-43. 10 out of the 16 FEE64 modules showed sub 20 kHz rates.

19.07 We turned off the power supply on the base of the snout support. PN300 Attachment 23. Rates did not change - attachment 44.

19.10 We turned off the mesytec PSU that is located in the bPlast NIM crate. No change was observed in the rates - attachment 45.

19.13 We turned off the R&SRMP4040 PSU that is located above the AIDA crate. And the noise situation did not change attachment 46.

19.22 Leaving the R&SRMP4040 PSU off we turned back on the mesytec PSU and PN300 PSU, attachment 47.

Summary:

It is clear that with no continuity on the short side we are able to reduce the noise back to the scenario where bPlast was not connected. However, powering bPlast introduced substantial noise in both DSSSDs that did not go away when turning all the PSUs off. There might be some hysteresis in the system (this is just speculation).

The situation is still fair considering that 10 out of the 16 FEE64 modules are in the sub 20 kHz rate level (good noise condition).

TO-DO for 11.04.2024

- Try bringing bPlast ground back to the PSU ground.

- Recheck the downstream detector bias and ground scheme.
       
Attachment 1: Screenshot_from_2024-04-10_09-43-13.png
Screenshot_from_2024-04-10_09-43-13.png
Attachment 2: Screenshot_from_2024-04-10_09-46-53.png
Screenshot_from_2024-04-10_09-46-53.png
Attachment 3: Screenshot_from_2024-04-10_09-49-03.png
Screenshot_from_2024-04-10_09-49-03.png
Attachment 4: Screenshot_from_2024-04-10_09-49-35.png
Screenshot_from_2024-04-10_09-49-35.png
Attachment 5: Screenshot_from_2024-04-10_09-50-23.png
Screenshot_from_2024-04-10_09-50-23.png
Attachment 6: Screenshot_from_2024-04-10_09-50-52.png
Screenshot_from_2024-04-10_09-50-52.png
Attachment 7: Screenshot_from_2024-04-10_09-50-42.png
Screenshot_from_2024-04-10_09-50-42.png
Attachment 8: Screenshot_from_2024-04-10_09-53-41.png
Screenshot_from_2024-04-10_09-53-41.png
Attachment 9: Screenshot_from_2024-04-10_09-53-17.png
Screenshot_from_2024-04-10_09-53-17.png
Attachment 10: Screenshot_from_2024-04-10_09-52-49.png
Screenshot_from_2024-04-10_09-52-49.png
Attachment 11: Screenshot_from_2024-04-10_09-52-22.png
Screenshot_from_2024-04-10_09-52-22.png
Attachment 12: Screenshot_from_2024-04-10_09-59-02.png
Screenshot_from_2024-04-10_09-59-02.png
Attachment 13: Screenshot_from_2024-04-10_10-51-20.png
Screenshot_from_2024-04-10_10-51-20.png
Attachment 14: Screenshot_from_2024-04-10_10-50-37.png
Screenshot_from_2024-04-10_10-50-37.png
Attachment 15: Screenshot_from_2024-04-10_10-52-40.png
Screenshot_from_2024-04-10_10-52-40.png
Attachment 16: Screenshot_from_2024-04-10_10-52-16.png
Screenshot_from_2024-04-10_10-52-16.png
Attachment 17: Screenshot_from_2024-04-10_11-41-42.png
Screenshot_from_2024-04-10_11-41-42.png
Attachment 18: IMG_0010.JPG
IMG_0010.JPG
Attachment 19: IMG_0007.JPG
IMG_0007.JPG
Attachment 20: IMG_0012.JPG
IMG_0012.JPG
Attachment 21: IMG_0005.JPG
IMG_0005.JPG
Attachment 22: IMG_0009.JPG
IMG_0009.JPG
Attachment 23: IMG_0015.JPG
IMG_0015.JPG
Attachment 24: IMG_0014.JPG
IMG_0014.JPG
Attachment 25: IMG_0017.JPG
IMG_0017.JPG
Attachment 26: IMG_0006.JPG
IMG_0006.JPG
Attachment 27: IMG_0013.JPG
IMG_0013.JPG
Attachment 28: Screenshot_from_2024-04-10_13-18-09.png
Screenshot_from_2024-04-10_13-18-09.png
Attachment 29: Screenshot_from_2024-04-10_15-55-15.png
Screenshot_from_2024-04-10_15-55-15.png
Attachment 30: Screenshot_from_2024-04-10_17-19-01.png
Screenshot_from_2024-04-10_17-19-01.png
Attachment 31: Screenshot_from_2024-04-10_17-13-25.png
Screenshot_from_2024-04-10_17-13-25.png
Attachment 32: Screenshot_from_2024-04-10_17-12-33.png
Screenshot_from_2024-04-10_17-12-33.png
Attachment 33: Screenshot_from_2024-04-10_17-04-11.png
Screenshot_from_2024-04-10_17-04-11.png
Attachment 34: Screenshot_from_2024-04-10_18-49-54.png
Screenshot_from_2024-04-10_18-49-54.png
Attachment 35: Screenshot_from_2024-04-10_18-49-50.png
Screenshot_from_2024-04-10_18-49-50.png
Attachment 36: Screenshot_from_2024-04-10_18-49-44.png
Screenshot_from_2024-04-10_18-49-44.png
Attachment 37: Screenshot_from_2024-04-10_19-00-27.png
Screenshot_from_2024-04-10_19-00-27.png
Attachment 38: Screenshot_from_2024-04-10_18-59-41.png
Screenshot_from_2024-04-10_18-59-41.png
Attachment 39: Screenshot_from_2024-04-10_18-59-26.png
Screenshot_from_2024-04-10_18-59-26.png
Attachment 40: Screenshot_from_2024-04-10_18-58-43.png
Screenshot_from_2024-04-10_18-58-43.png
Attachment 41: Screenshot_from_2024-04-10_18-58-21.png
Screenshot_from_2024-04-10_18-58-21.png
Attachment 42: Screenshot_from_2024-04-10_18-57-53.png
Screenshot_from_2024-04-10_18-57-53.png
Attachment 43: Screenshot_from_2024-04-10_18-57-46.png
Screenshot_from_2024-04-10_18-57-46.png
Attachment 44: Screenshot_from_2024-04-10_19-23-16.png
Screenshot_from_2024-04-10_19-23-16.png
Attachment 45: Screenshot_from_2024-04-10_19-20-55.png
Screenshot_from_2024-04-10_19-20-55.png
Attachment 46: Screenshot_from_2024-04-10_19-12-22.png
Screenshot_from_2024-04-10_19-12-22.png
Attachment 47: Screenshot_from_2024-04-10_19-09-07.png
Screenshot_from_2024-04-10_19-09-07.png
  569   Tue Apr 9 13:22:41 2024 TD, JB, CCContinued of 9th of April

Disconnected test - pulser from aida08. Noise conditions improved slightly, probably less pulser events. (Attachments 1 & 2).

We recabled the dew point sensor and flow sensor for the water supply interlock to the FEE64s. When we tried to power up the interlock box, the PSU (charging block) LED was blinking indicating an internal power instability. We then plugged each of the sensors out and into the interlock box and found that the flow sensor was the source of the power supply issue.

The HDMI cable of aida04 appears to be broken or wobbly, but the signals still look the same after powering up again. The adaptor board was reseated.

Pulser peak of test - remeasured the peak in aida02 1.8.L and found the same result as before with 118 channels FWHM c. 95-100 keV FWHM energy resolution. (Attachment 3)

bPlast booster boards connected caused dramatic increase in the noise observed in AIDA. Grounding still needs to take place.

TO-DO for 10.04 - 11.04:

- aida04 asic #1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?

- noise optimisation of aida-bplast -- (friday if BB7 also powered).

 

Attachment 1: Screenshot_from_2024-04-09_14-22-20.png
Screenshot_from_2024-04-09_14-22-20.png
Attachment 2: Screenshot_from_2024-04-09_14-22-16.png
Screenshot_from_2024-04-09_14-22-16.png
Attachment 3: Screenshot_from_2024-04-09_17-18-45.png
Screenshot_from_2024-04-09_17-18-45.png
  568   Tue Apr 9 09:04:07 2024 TD[How To] Remote control of BNC PB-5
BNC PB-5 set to remote control

Login to RPi nnrpi2

ssh -X nnrpi2 -l pi

Start PB-5 app - attachment 1

./BNC-PB-5/pb5.py
Attachment 1: Screenshot_from_2024-04-09_10-03-42.png
Screenshot_from_2024-04-09_10-03-42.png
  567   Tue Apr 9 08:37:47 2024 TDTuesday 9 April
09.36 Cooling water temperature and flow OK

      FEE64 power ON

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida03 WR decoder status - attachment 3

      WR timestamps OK - attachment 4

      info code 4, correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 5-10

      per FEE64 Rate spectra - attachments 11

      per 1.8.W spectra - 20us FSR - attachments 12-13

12.25 DSSSD bias & leakage current OK - attachment 16

      FEE64 temps OK - attachment 15
      *except* aida02 ASIC temp which is known to be u/s

      ADC data item stats - attachments 14

      system stable
Attachment 1: Screenshot_from_2024-04-09_09-50-58.png
Screenshot_from_2024-04-09_09-50-58.png
Attachment 2: Screenshot_from_2024-04-09_09-51-07.png
Screenshot_from_2024-04-09_09-51-07.png
Attachment 3: Screenshot_from_2024-04-09_09-51-26.png
Screenshot_from_2024-04-09_09-51-26.png
Attachment 4: Screenshot_from_2024-04-09_09-51-56.png
Screenshot_from_2024-04-09_09-51-56.png
Attachment 5: Screenshot_from_2024-04-09_09-53-31.png
Screenshot_from_2024-04-09_09-53-31.png
Attachment 6: Screenshot_from_2024-04-09_09-53-19.png
Screenshot_from_2024-04-09_09-53-19.png
Attachment 7: Screenshot_from_2024-04-09_09-52-58.png
Screenshot_from_2024-04-09_09-52-58.png
Attachment 8: Screenshot_from_2024-04-09_09-52-48.png
Screenshot_from_2024-04-09_09-52-48.png
Attachment 9: Screenshot_from_2024-04-09_09-52-38.png
Screenshot_from_2024-04-09_09-52-38.png
Attachment 10: Screenshot_from_2024-04-09_09-52-25.png
Screenshot_from_2024-04-09_09-52-25.png
Attachment 11: Screenshot_from_2024-04-09_09-55-57.png
Screenshot_from_2024-04-09_09-55-57.png
Attachment 12: Screenshot_from_2024-04-09_09-54-17.png
Screenshot_from_2024-04-09_09-54-17.png
Attachment 13: Screenshot_from_2024-04-09_09-53-45.png
Screenshot_from_2024-04-09_09-53-45.png
Attachment 14: Screenshot_from_2024-04-09_12-24-55.png
Screenshot_from_2024-04-09_12-24-55.png
Attachment 15: Screenshot_from_2024-04-09_12-24-43.png
Screenshot_from_2024-04-09_12-24-43.png
Attachment 16: Screenshot_from_2024-04-09_12-24-38.png
Screenshot_from_2024-04-09_12-24-38.png
  566   Mon Apr 8 16:39:00 2024 JB, CC, TDMonday 8 April
17.32 Power and detector bias cycle

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2
       *except* aida02 ASIC temp u/s

      ASIC settings 2024Mar27-11.25.32
       slow comparator p+n FEE64s 0xa (100keV), n+n FEE64s 0xf (150keV)

      BNC PB-5 pulser
      amplitude 10.0V
      attenuation x10
      frequency 25Hz
      polarity -
      tau_d 1ms
      tail pulse

      test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

      test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

      All system wide checks OK *except* aida02 & aida03 WR decoder status errors - attachment 10

      WR timestamps OK - attachment 11

      ADC data item stats - attachment 3
       12 of 16 < 20k, all < 100k

      per FEE64 Rate spectra - attachment 4

      per n+n FEE64 1.8.L spectra - attachment 5

      1.8.W spectra - 20us FSR - attachments 6-9
       preamplifier output noise generally very good

*Current ground configuration*

CAEN N1419ET LK fitted (non floating outputs)

ground cable from aida04-aida12-aida02-aida09-aida01-aida05
HV#0 aida12-aida03-aida15

ground cable from aida08-aida16-aida06-aida10-aida14-aida13
HV#1 aida16-aida07-aida11

LK3 fitted aida03, aida07

LK1 fitted aida02, aida04, aid06, aida08

test - distributed by daisy chain to n+n FEE64s - chain terminated by 50 Ohm - currently connected to pulser

test + distributed by daisy chain to p+n FEE64s - chain terminated by 50 Ohm - currently disconnected from pulser

all DSSSD ribbon cable drain wires grounded to their resepctive AIDA adaptor PCBs

JB observed open circuit between AIDA Al snout and AIDA support frame ( as expected )

AIDA PSU cabling as reported https://elog.ph.ed.ac.uk/DESPEC/560

Snout configuration:

- all bPlast cables are disconnected and floating. 
- CC taped up all the cables and exit points in snout.
- CC, JB covered the exit of the snout with two additional layers of aluminium foil which were also taped shut.


To Do:

- repower detector, see if similar conditions persist.
- disconnect test - pulser, again see if conditions persist.

- aida04 asic#1 investigate noise
- aida14 asic #1 & asic #2 u/s? replace ASIC mezzanine?




       
Attachment 1: Screenshot_from_2024-04-08_17-32-49.png
Screenshot_from_2024-04-08_17-32-49.png
Attachment 2: Screenshot_from_2024-04-08_17-33-10.png
Screenshot_from_2024-04-08_17-33-10.png
Attachment 3: Screenshot_from_2024-04-08_17-33-14.png
Screenshot_from_2024-04-08_17-33-14.png
Attachment 4: Screenshot_from_2024-04-08_17-33-33.png
Screenshot_from_2024-04-08_17-33-33.png
Attachment 5: Screenshot_from_2024-04-08_17-34-40.png
Screenshot_from_2024-04-08_17-34-40.png
Attachment 6: Screenshot_from_2024-04-08_17-35-00.png
Screenshot_from_2024-04-08_17-35-00.png
Attachment 7: Screenshot_from_2024-04-08_17-35-19.png
Screenshot_from_2024-04-08_17-35-19.png
Attachment 8: Screenshot_from_2024-04-08_17-37-28.png
Screenshot_from_2024-04-08_17-37-28.png
Attachment 9: Screenshot_from_2024-04-08_17-37-56.png
Screenshot_from_2024-04-08_17-37-56.png
Attachment 10: Screenshot_from_2024-04-08_17-48-23.png
Screenshot_from_2024-04-08_17-48-23.png
Attachment 11: Screenshot_from_2024-04-08_17-48-49.png
Screenshot_from_2024-04-08_17-48-49.png
  565   Mon Apr 8 11:18:19 2024 TDS505 offline analysis data file R3_150
DRG quotes ( from Elog ) S505 FEE64 configuration as

"Configuration: Single/Narrow - 8 FEEs, 2 DSSDs

DSSD 1:
 X 0-63    =  3 (HV -ve)
 X 64-127  =  1
 Y 0-63    =  4 (HV 0v) 
 Y 64-127  =  2 
DSSD 2:
 X 0-63    =  7 (HV -ve)
 X 64-127  =  5
 Y 0-63    =  8 (HV 0v)
 Y 64-127  =  6 


Scalers:
1 - Pulser
2 - n/c
3 - Time Machine Original
4 - Time Machine Delayed
5 - n/c
6 - n/c
7 - SC41 L
8 - SC41 R"

LEC (20MeV FSR) spectra

attachment 1 m_p versus m_n DSSSD#1 & DSSSD#2
attachment 2 E_p versus E_n DSSSD#1 & DSSSD#2

HEC (20GeV FSR) spectra

attachment 3 m_p versus m_n DSSSD#1 & DSSSD#2
attachment 4 E_p versus E_n DSSSD#1 & DSSSD#2

attachment variables.dat ( ADC offsets, e_diff windows & FEE64 onfiguration ) NAMELIST I/O format
Attachment 1: Screenshot_from_2024-03-29_20-21-07.png
Screenshot_from_2024-03-29_20-21-07.png
Attachment 2: Screenshot_from_2024-03-29_20-21-45.png
Screenshot_from_2024-03-29_20-21-45.png
Attachment 3: Screenshot_from_2024-03-29_20-22-29.png
Screenshot_from_2024-03-29_20-22-29.png
Attachment 4: Screenshot_from_2024-03-29_20-23-06.png
Screenshot_from_2024-03-29_20-23-06.png
Attachment 5: variables.dat
  $variables
  offset(   0) =    28.32
  offset(   1) =    17.71
  offset(   2) =    -9.75
  offset(   3) =    -5.24
  offset(   4) =   -10.84
  offset(   5) =     0.13
  offset(   6) =    28.59
  offset(   7) =    28.54
  offset(   8) =     0.83
  offset(   9) =   -12.15
  offset(  10) =    29.36
  offset(  11) =   -50.47
  offset(  12) =    10.03
  offset(  13) =   -26.90
  offset(  14) =    12.49
  offset(  15) =    10.57
  offset(  16) =     9.29
  offset(  17) =     1.09
  offset(  18) =    21.86
  offset(  19) =    -0.32
  offset(  20) =    22.59
  offset(  21) = -9999.99
  offset(  22) =   -20.22
  offset(  23) =     6.61
  offset(  24) =    -8.80
  offset(  25) =    -1.15
  offset(  26) =    13.08
  offset(  27) =    -9.03
  offset(  28) =    -2.67
  offset(  29) =    -9.42
  offset(  30) =     8.07
  offset(  31) =    19.01
  offset(  32) =     6.86
  offset(  33) =    15.16
  offset(  34) =    -3.89
  offset(  35) =   -18.80
  offset(  36) =    17.79
  offset(  37) =    16.45
  offset(  38) =    -0.74
  offset(  39) =     0.69
  offset(  40) =   -14.59
  offset(  41) =   -10.02
  offset(  42) =    11.38
  offset(  43) =    -2.57
  offset(  44) =    15.48
  offset(  45) =    14.57
  offset(  46) =     2.95
  offset(  47) =    30.70
  offset(  48) =   -12.55
  offset(  49) =   -15.61
  offset(  50) =    -6.61
  offset(  51) =    -7.10
  offset(  52) =   -33.20
  offset(  53) =    21.42
  offset(  54) =    -4.07
  offset(  55) =   -18.23
  offset(  56) =     8.83
  offset(  57) =   -22.74
  offset(  58) =    11.47
  offset(  59) =    10.67
  offset(  60) =   -28.65
  offset(  61) =    15.29
  offset(  62) =     5.76
  offset(  63) =   -11.57
  offset(  64) =     1.95
  offset(  65) = -9999.99
  offset(  66) =    22.52
  offset(  67) =    42.10
  offset(  68) =    -0.20
  offset(  69) =     9.78
  offset(  70) =   -11.48
  offset(  71) =    -0.03
  offset(  72) =   -12.50
  offset(  73) =    25.08
  offset(  74) =    12.77
  offset(  75) =    -2.76
  offset(  76) =     7.90
  offset(  77) =     9.85
  offset(  78) =     5.20
  offset(  79) =   -13.91
  offset(  80) =    18.82
  offset(  81) =    -8.25
  offset(  82) =    -8.40
  offset(  83) =    13.67
  offset(  84) =   -10.37
  offset(  85) =    28.32
  offset(  86) =    11.42
  offset(  87) =     2.74
  offset(  88) =   -24.88
  offset(  89) =    11.28
  offset(  90) =   -26.69
  offset(  91) =     0.91
  offset(  92) =     8.04
  offset(  93) =     7.64
  offset(  94) =    10.10
  offset(  95) =   -20.71
  offset(  96) =     6.72
  offset(  97) =   -12.63
  offset(  98) =    10.27
  offset(  99) =    16.05
  offset( 100) =     8.96
  offset( 101) =    33.57
  offset( 102) =    28.81
  offset( 103) =    23.49
  offset( 104) =     7.96
  offset( 105) =     6.07
  offset( 106) =    21.57
  offset( 107) =    13.06
  offset( 108) =     0.82
  offset( 109) =    -3.55
  offset( 110) =    -5.21
  offset( 111) =    11.48
  offset( 112) =    16.21
  offset( 113) =    -0.10
  offset( 114) =     3.87
  offset( 115) =    19.76
  offset( 116) =    -9.11
  offset( 117) =    24.64
  offset( 118) =     1.82
  offset( 119) = -9999.99
  offset( 120) =    22.80
  offset( 121) =    -7.37
  offset( 122) =   -18.32
  offset( 123) =   -24.74
  offset( 124) =    -4.71
  offset( 125) =    19.64
  offset( 126) =    53.61
  offset( 127) =     6.57
  offset( 128) =     3.21
  offset( 129) =     7.20
  offset( 130) =     5.55
  offset( 131) =    -3.40
  offset( 132) =   -16.40
  offset( 133) =    -1.71
  offset( 134) =   -14.85
  offset( 135) =   -29.53
  offset( 136) =     0.68
  offset( 137) =    -0.41
  offset( 138) =    17.59
  offset( 139) =   -28.92
  offset( 140) =    -8.79
  offset( 141) =    11.69
  offset( 142) =    -5.80
  offset( 143) =    -8.21
  offset( 144) =    13.90
  offset( 145) =    11.27
  offset( 146) =   -17.02
  offset( 147) =    -8.19
  offset( 148) =     6.14
  offset( 149) =   -13.13
  offset( 150) =    15.41
  offset( 151) =     0.12
  offset( 152) =    -5.02
  offset( 153) =     4.25
  offset( 154) =     3.84
  offset( 155) =    13.34
  offset( 156) =    16.79
  offset( 157) =   -36.21
  offset( 158) =    -2.18
  offset( 159) =    12.94
  offset( 160) =     2.05
  offset( 161) =   -17.31
  offset( 162) =    18.38
  offset( 163) =    -7.17
  offset( 164) =     6.59
  offset( 165) =   -13.36
  offset( 166) =   -12.11
  offset( 167) =    -3.91
  offset( 168) =    10.97
  offset( 169) =   -25.72
  offset( 170) =   -11.53
  offset( 171) =     1.94
  offset( 172) =   -12.76
  offset( 173) =    18.55
  offset( 174) =    13.49
  offset( 175) =    -9.45
  offset( 176) =   -10.19
  offset( 177) =    18.69
  offset( 178) =    24.49
  offset( 179) =    30.44
  offset( 180) =    41.95
  offset( 181) =     1.55
  offset( 182) =    20.88
  offset( 183) =     4.76
  offset( 184) =    15.80
  offset( 185) =    12.37
  offset( 186) =   -13.03
  offset( 187) =    15.64
  offset( 188) =    -8.44
  offset( 189) =    -3.02
  offset( 190) =    10.57
  offset( 191) =     6.88
  offset( 192) =   -17.81
  offset( 193) =    -5.77
  offset( 194) =   -18.29
  offset( 195) =    17.62
  offset( 196) =   -30.66
  offset( 197) =    -7.37
  offset( 198) =    36.32
  offset( 199) =     3.42
  offset( 200) =   -21.92
  offset( 201) =    15.88
  offset( 202) =   -17.33
  offset( 203) =    -2.71
  offset( 204) =   -21.06
  offset( 205) =    -4.53
  offset( 206) =    -1.59
  offset( 207) =    -9.54
  offset( 208) =    -0.04
  offset( 209) =    23.57
  offset( 210) =    13.83
  offset( 211) =    -9.28
  offset( 212) =    12.42
  offset( 213) =    23.85
  offset( 214) =     7.90
  offset( 215) =    -7.29
  offset( 216) =    26.18
  offset( 217) =    33.05
  offset( 218) =    32.10
  offset( 219) =    -8.10
  offset( 220) =    16.16
  offset( 221) =     3.94
  offset( 222) =   -18.22
  offset( 223) =   -16.84
  offset( 224) =     5.79
  offset( 225) =    21.07
  offset( 226) =    45.18
  offset( 227) =   -13.85
  offset( 228) =     6.76
  offset( 229) =    13.59
  offset( 230) =     8.28
  offset( 231) =    -9.32
  offset( 232) =    21.30
  offset( 233) =     7.02
  offset( 234) =    24.11
  offset( 235) =    16.26
  offset( 236) =    11.46
  offset( 237) =     2.16
  offset( 238) =     7.19
  offset( 239) =    18.45
  offset( 240) =     6.65
  offset( 241) =     6.56
  offset( 242) =   -33.59
  offset( 243) =    14.71
  offset( 244) =    21.28
  offset( 245) =    -0.73
  offset( 246) =   -10.29
  offset( 247) =    28.50
  offset( 248) =   -11.37
  offset( 249) =   -18.92
  offset( 250) =    23.12
  offset( 251) =     8.87
  offset( 252) =     9.64
  offset( 253) =    16.43
  offset( 254) =    -2.70
  offset( 255) =    26.14
  offset( 256) =    -4.99
  offset( 257) =   -10.88
  offset( 258) =    27.41
  offset( 259) =    -3.48
  offset( 260) =   -11.18
  offset( 261) =    12.60
  offset( 262) =    22.19
  offset( 263) =   -20.16
  offset( 264) =    20.32
  offset( 265) =     9.66
  offset( 266) =    13.30
  offset( 267) =   -12.64
  offset( 268) =    53.69
  offset( 269) =    23.15
  offset( 270) =    43.60
  offset( 271) =    48.58
  offset( 272) =   -13.12
  offset( 273) =   -12.57
  offset( 274) =    35.19
  offset( 275) =    14.84
  offset( 276) =    33.05
  offset( 277) =    13.73
  offset( 278) =    45.45
  offset( 279) =    55.49
  offset( 280) =    48.13
  offset( 281) =     3.86
  offset( 282) =    28.44
  offset( 283) =    23.73
  offset( 284) =    11.53
  offset( 285) =   -19.46
  offset( 286) =    10.81
  offset( 287) =    17.23
  offset( 288) =   -13.85
  offset( 289) =    30.17
  offset( 290) =    10.70
  offset( 291) =    28.06
  offset( 292) =   -14.35
  offset( 293) =    23.46
  offset( 294) =    15.16
  offset( 295) =   -25.15
  offset( 296) =   -15.03
  offset( 297) =   -24.74
  offset( 298) =    -5.94
... 228 more lines ...
  564   Mon Apr 8 10:23:16 2024 TD, JBAIDA Noise optimisation

In the morning we reinstalled FEE64s aida06, aida13, aida07. The rails of the FEE64 boards were found to be incorrect (that of the rails that were used in CARME).

After replacing the rails on the abovementioned FEE64s we powercycled AIDA and found the following results in the histograms and statistics.

As can be seen in attachement 4, aida16 has a noisy waveform and we are going to investigate this further.

When setting the threshold to 0x64 (1 MeV) on the slow comparator threshold we saw that the low energy branch of aida04 was still noisy. (Attachment 6)

- Setting the slow comparator threshold to 0x32 (0.5 MeV) introduced noise into aida08, aida16 and substantially more noise in aida04. (Attachment 7)

- Setting the slow comparator threshold to 0x14 (0.2 MeV) introduced noise into aida11 and substantially more noise in aida04, aida08, aida16 and aida06. (Attachment 8)

- Setting the slow comparator threshold to 0xf (0.15 MeV) introduced more noise into the same aforementioned channels aida04, aida06, aida08, aida11, aida16 (additionally aida02), barring hot strips. (Attachment 9)

 

We covered the end of the snout with a black cloth and this reduced the noise in some of the channels (Attachment 10 and 11).

- We covered the end of the snout with two layers of aluminium foil - this produced the same result as covering the snout with the black cloth (Attachment 12, 13, 14 & 15).

We resumed after lunch.

- Replaced the aida16 adaptor PCB with a spare, and this produced a much more sensible rate and hitpattern in aida16, which could lead us to believe that the aida adapter card was faulty or misaligned. (Attachment 16, 17, 18 & 19).

- Placed two LK1 back on the n+n side of both DSSSDs on aida06 and aida02. The noise and rate did not change at all. (Attachment 20 & 21)

Pulser tests:

- The test + pulser was connected to the p+n sides (top and bottom) at 25 Hz.

- After checking the ASIC control 2 times over, the pulser seems to be ok for all p+n channels excluding aida01, aida13 and aida14. (Attachment 23)

- The pulser peak width of aida09 1.8.L was found to be c. 55 channels FWHM on uncalibrated scale which corresponds to 35 - 40 keV FWHM in energy resolution. (Attachment 24)

- After reseating the adaptor board PCBs we found the same result as before reseating the adaptor boards (Attachment 25).

- Connecting the test - pulser to the n+n  side we found that most of the ASICs behaved normally which ASIC 1 in aida04 was showing abnormal behaviour. (Attachment 26, 28). The test + pulser loop is still in place.

-  The pulser peak width of aida02 1.8.L was found to be c. 118 channels FWHM on uncalibrated scale which corresponds to 95 - 100 keV FWHM in energy resolution. (Attachment 27)

- We set the thresholds on the n+n side to 0xf and the p+n side to 0xa. Subsequently, we observed improved noise conditions in many of the channels. The still noisy channels are aida01, aida04, aida08. (Attachment 29, 30). This might be because we have introduced an addtional ground loop with the pusler tests, but could also mean that the ground loop that is currently being used is not sufficient or poorly implemented.

 

Summary:

- From the pulser test it is clear that aida14 ASICs 1&2 are not in good shape and appear to be faulty. 

- We managed to improve from 1 aida module in the sub 20 kHz rate range to 6 aida modules in the sub 20 kHz rate range by the end of the tests.

- We set the slow comparator thresholds on the n+n side to 0xf and the p+n side to 0xa. Subsequently, we observed improved noise conditions in many of the channels. The still noisy channels are aida01, aida04, aida08. (Attachment 29, 30). This might be because we have introduced an additional ground loop with the pusler tests, but could also mean that the ground loop that is currently being used is not sufficient or poorly implemented.

 

Attachment 1: Screenshot_from_2024-04-08_11-28-11.png
Screenshot_from_2024-04-08_11-28-11.png
Attachment 2: Screenshot_from_2024-04-08_11-27-25.png
Screenshot_from_2024-04-08_11-27-25.png
Attachment 3: Screenshot_from_2024-04-08_11-26-38.png
Screenshot_from_2024-04-08_11-26-38.png
Attachment 4: Screenshot_from_2024-04-08_11-26-31.png
Screenshot_from_2024-04-08_11-26-31.png
Attachment 5: Screenshot_from_2024-04-08_11-25-57.png
Screenshot_from_2024-04-08_11-25-57.png
Attachment 6: Screenshot_from_2024-04-08_11-31-34.png
Screenshot_from_2024-04-08_11-31-34.png
Attachment 7: Screenshot_from_2024-04-08_11-34-40.png
Screenshot_from_2024-04-08_11-34-40.png
Attachment 8: Screenshot_from_2024-04-08_11-36-50.png
Screenshot_from_2024-04-08_11-36-50.png
Attachment 9: Screenshot_from_2024-04-08_11-40-33.png
Screenshot_from_2024-04-08_11-40-33.png
Attachment 10: Screenshot_from_2024-04-08_11-51-46.png
Screenshot_from_2024-04-08_11-51-46.png
Attachment 11: Screenshot_from_2024-04-08_11-51-50.png
Screenshot_from_2024-04-08_11-51-50.png
Attachment 12: Screenshot_from_2024-04-08_12-09-33.png
Screenshot_from_2024-04-08_12-09-33.png
Attachment 13: Screenshot_from_2024-04-08_12-09-37.png
Screenshot_from_2024-04-08_12-09-37.png
Attachment 14: Screenshot_from_2024-04-08_12-10-20.png
Screenshot_from_2024-04-08_12-10-20.png
Attachment 15: Screenshot_from_2024-04-08_12-10-57.png
Screenshot_from_2024-04-08_12-10-57.png
Attachment 16: Screenshot_from_2024-04-08_15-09-34.png
Screenshot_from_2024-04-08_15-09-34.png
Attachment 17: Screenshot_from_2024-04-08_15-09-08.png
Screenshot_from_2024-04-08_15-09-08.png
Attachment 18: Screenshot_from_2024-04-08_14-45-49.png
Screenshot_from_2024-04-08_14-45-49.png
Attachment 19: Screenshot_from_2024-04-08_14-45-05.png
Screenshot_from_2024-04-08_14-45-05.png
Attachment 20: Screenshot_from_2024-04-08_14-44-19.png
Screenshot_from_2024-04-08_14-44-19.png
Attachment 21: Screenshot_from_2024-04-08_14-44-14.png
Screenshot_from_2024-04-08_14-44-14.png
Attachment 22: Screenshot_from_2024-04-08_16-03-19.png
Screenshot_from_2024-04-08_16-03-19.png
Attachment 23: Screenshot_from_2024-04-08_16-07-48.png
Screenshot_from_2024-04-08_16-07-48.png
Attachment 24: Screenshot_from_2024-04-08_16-13-53.png
Screenshot_from_2024-04-08_16-13-53.png
Attachment 25: Screenshot_from_2024-04-08_16-38-15.png
Screenshot_from_2024-04-08_16-38-15.png
Attachment 26: Screenshot_from_2024-04-08_16-48-25.png
Screenshot_from_2024-04-08_16-48-25.png
Attachment 27: Screenshot_from_2024-04-08_16-52-17.png
Screenshot_from_2024-04-08_16-52-17.png
Attachment 28: Screenshot_from_2024-04-08_16-50-12.png
Screenshot_from_2024-04-08_16-50-12.png
Attachment 29: Screenshot_from_2024-04-08_17-00-38.png
Screenshot_from_2024-04-08_17-00-38.png
Attachment 30: Screenshot_from_2024-04-08_16-58-25.png
Screenshot_from_2024-04-08_16-58-25.png
  563   Sun Apr 7 18:13:56 2024 TDSunday 7 April contd.
19.10 CAEN N1419ET LK fitted
      power cycle

attachments 1-6
Attachment 1: Screenshot_from_2024-04-07_19-13-19.png
Screenshot_from_2024-04-07_19-13-19.png
Attachment 2: Screenshot_from_2024-04-07_19-12-40.png
Screenshot_from_2024-04-07_19-12-40.png
Attachment 3: Screenshot_from_2024-04-07_19-11-24.png
Screenshot_from_2024-04-07_19-11-24.png
Attachment 4: Screenshot_from_2024-04-07_19-11-09.png
Screenshot_from_2024-04-07_19-11-09.png
Attachment 5: Screenshot_from_2024-04-07_19-10-45.png
Screenshot_from_2024-04-07_19-10-45.png
Attachment 6: Screenshot_from_2024-04-07_19-10-30.png
Screenshot_from_2024-04-07_19-10-30.png
  562   Sun Apr 7 13:08:52 2024 TDFEE64 configuration
FEE64 configuration per https://elog.gsi.de/despec/S100/6

Configuration: Wide - 16 FEEs, 2 DSSD

DSSD 1:
 X 0-63    =  15 
 X 63-127  =  9
 X 128-191 =  3 
 X 192-255 =  1
 X 256-319 =  12 
 X 320-385 =  5
 Y 0-63    =  4 
 Y 63-127  =  2 


DSSD 2:
 X 0-63    =  11 
 X 63-127  =  10
 X 128-191 =  7
 X 192-255 =  14
 X 256-319 =  16 
 X 320-385 =  13
 Y 0-63    =  8
 Y 63-127  =  6 

Scalers:
1 - Pulser
2 - FRS Accepted
3 - Time Machine Undelayed
4 - Time Machine Delayed
5 - SC41 L
6 - SC41 R


Thresholds (Slow / Fast / Implant)
DSSD 1 p+n : 0xA (100 keV) / 0xFF (disabled) / 0x02 (200 MeV)
DSSD 1 n+n : 0xF (150 keV) / 0xFF (disabled) / 0x02 (200 MeV)
Attachment 1: AIDA_S100_WiringPlan.drawio.png
AIDA_S100_WiringPlan.drawio.png
  561   Sun Apr 7 12:16:27 2024 TDaida06 system console log
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:07:47|
Finished copying zImage to RAM 
07:04:24/13:07:48|
Found 0 errors checking kernel image
07:04:24/13:07:49|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
07:04:24/13:07:49|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
07:04:24/13:07:49|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
07:04:24/13:07:49|SPIconfig done now to set up the LMK3200 registers
07:04:24/13:07:49|LMK #0 : regInit[0]=0x80000000
07:04:24/13:07:49|LMK #0 : regInit[1]=0x10070600
07:04:24/13:07:49|LMK #0 : regInit[2]=0x60601
07:04:24/13:07:50|LMK #0 : regInit[3]=0x60602
07:04:24/13:07:50|LMK #0 : regInit[4]=0x60603
07:04:24/13:07:50|LMK #0 : regInit[5]=0x70624
07:04:24/13:07:50|LMK #0 : regInit[6]=0x70605
07:04:24/13:07:50|LMK #0 : regInit[7]=0x70606
07:04:24/13:07:50|LMK #0 : regInit[8]=0x70627
07:04:24/13:07:50|LMK #0 : regInit[9]=0x10000908
07:04:24/13:07:50|LMK #0 : regInit[10]=0xA0022A09
07:04:24/13:07:50|LMK #0 : regInit[11]=0x82800B
07:04:24/13:07:50|LMK #0 : regInit[12]=0x28C800D
07:04:24/13:07:50|LMK #0 : regInit[13]=0x830020E
07:04:24/13:07:50|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
07:04:24/13:07:50|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
07:04:24/13:07:50|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
07:04:24/13:07:50|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal not completed 
ADC calibrate failed 
Jumping to kernel simpleboot...
07:04:24/13:07:51|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
07:04:24/13:07:54|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
07:04:24/13:08:00|Using Xilinx Virtex440 machine description
07:04:24/13:08:01|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
07:04:24/13:08:01|Zone PFN ranges:
07:04:24/13:08:01|  DMA      0x00000000 -> 0x00007000
07:04:24/13:08:01|  Normal   0x00007000 -> 0x00007000
07:04:24/13:08:01|Movable zone start PFN for each node
07:04:24/13:08:01|early_node_map[1] active PFN ranges
07:04:24/13:08:01|    0: 0x00000000 -> 0x00007000
07:04:24/13:08:01|MMU: Allocated 1088 bytes of context maps for 255 contexts
07:04:24/13:08:01|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
07:04:24/13:08:02|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
07:04:24/13:08:02|PID hash table entries: 512 (order: 9, 2048 bytes)
07:04:24/13:08:02|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
07:04:24/13:08:02|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
07:04:24/13:08:02|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
07:04:24/13:08:02|Kernel virtual memory layout:
07:04:24/13:08:02|  * 0xffffe000..0xfffff000  : fixmap
07:04:24/13:08:02|  * 0xfde00000..0xfe000000  : consistent mem
07:04:24/13:08:02|  * 0xfde00000..0xfde00000  : early ioremap
07:04:24/13:08:02|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
07:04:24/13:08:02|NR_IRQS:512
07:04:24/13:08:02|clocksource: timebase mult[a00000] shift[22] registered
07:04:24/13:08:02|Console: colour dummy device 80x25
07:04:24/13:08:02|Mount-cache hash table entries: 512
07:04:24/13:08:02|NET: Registered protocol family 16
07:04:24/13:08:02|PCI: Probing PCI hardware
07:04:24/13:08:02|bio: create slab <bio-0> at 0
07:04:24/13:08:02|NET: Registered protocol family 2
07:04:24/13:08:02|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
07:04:24/13:08:02|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
07:04:24/13:08:03|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
07:04:24/13:08:03|TCP: Hash tables configured (established 4096 bind 4096)
07:04:24/13:08:03|TCP reno registered
07:04:24/13:08:03|NET: Registered protocol family 1
07:04:24/13:08:03|ROMFS MTD (C) 2007 Red Hat, Inc.
07:04:24/13:08:03|msgmni has been set to 214
07:04:24/13:08:03|io scheduler noop registered
07:04:24/13:08:03|io scheduler anticipatory registered
07:04:24/13:08:03|io scheduler deadline registered
07:04:24/13:08:03|io scheduler cfq registered (default)
07:04:24/13:08:03|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
07:04:24/13:08:03|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
07:04:24/13:08:03|console [ttyS0] enabled
07:04:24/13:08:03|brd: module loaded
07:04:24/13:08:03|loop: module loaded
07:04:24/13:08:03|Device Tree Probing 'ethernet'
07:04:24/13:08:03|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
07:04:24/13:08:03|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
07:04:24/13:08:03|XLlTemac: DCR address: 0x80
07:04:24/13:08:03|XLlTemac: buffer descriptor size: 32768 (0x8000)
07:04:24/13:08:03|XLlTemac: Allocating DMA descriptors with kmalloc
07:04:24/13:08:03|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
07:04:24/13:08:04|XTemac: PHY detected at address 7.
07:04:24/13:08:04|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
07:04:24/13:08:04|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:08:04|Using buffer write method
07:04:24/13:08:04|cfi_cmdset_0001: Erase suspend on write enabled
07:04:24/13:08:04|cmdlinepart partition parsing not available
07:04:24/13:08:04|RedBoot partition parsing not available
07:04:24/13:08:04|Creating 5 MTD partitions on "fc000000.flash":
07:04:24/13:08:04|0x000000000000-0x000000500000 : "golden_firmware"
07:04:24/13:08:04|0x000000500000-0x000000800000 : "golden_kernel"
07:04:24/13:08:04|0x000000800000-0x000000d00000 : "user_firmware"
07:04:24/13:08:04|0x000000d00000-0x000000fe0000 : "user_kernel"
07:04:24/13:08:04|0x000000fe0000-0x000001000000 : "env_variables"
07:04:24/13:08:05|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
07:04:24/13:08:05|SPI: XIlinx spi: bus number now 32766
07:04:24/13:08:05|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
07:04:24/13:08:05|SPI: XIlinx spi: bus number now 32765
07:04:24/13:08:05|mice: PS/2 mouse device common for all mice
07:04:24/13:08:05|Device Tree Probing 'i2c'
07:04:24/13:08:05| #0 at 0x81600000 mapped to 0xD1030000, irq=22
07:04:24/13:08:05|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
07:04:24/13:08:05|TCP cubic registered
07:04:24/13:08:05|NET: Registered protocol family 17
07:04:24/13:08:05|RPC: Registered udp transport module.
07:04:24/13:08:05|RPC: Registered tcp transport module.
07:04:24/13:08:05|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:08:06|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:08:06|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:08:06|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:08:08|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:08:08|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:08:08|Sending DHCP requests .
07:04:24/13:08:10|eth0: XLlTemac: PHY Link carrier lost.
07:04:24/13:08:10|..... timed out!
07:04:24/13:09:39|IP-Config: Reopening network devices...
07:04:24/13:09:39|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:09:40|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:09:40|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:09:40|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:09:42|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:09:42|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:09:42|Sending DHCP requests ...... timed out!
07:04:24/13:11:15|IP-Config: Auto-configuration of network failed.
07:04:24/13:11:15|Root-NFS: No NFS server available, giving up.
07:04:24/13:11:15|VFS: Unable to mount root fs via NFS, trying floppy.
07:04:24/13:11:15|VFS: Cannot open root device "nfs" or unknown-block(2,0)
07:04:24/13:11:15|Please append a correct "root=" boot option; here are the available partitions:
07:04:24/13:11:15|Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
07:04:24/13:11:15|Call Trace:
07:04:24/13:11:15|[c6827ed0] [c0005de8] show_stack+0x44/0x16c (unreliable)
07:04:24/13:11:15|[c6827f10] [c00345bc] panic+0x94/0x168
07:04:24/13:11:15|[c6827f60] [c0341d34] mount_block_root+0x12c/0x244
07:04:24/13:11:15|[c6827fb0] [c03420d8] prepare_namespace+0x17c/0x208
07:04:24/13:11:15|[c6827fd0] [c0341220] kernel_init+0x104/0x130
07:04:24/13:11:15|[c6827ff0] [c000e140] kernel_thread+0x4c/0x68
07:04:24/13:11:15|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS:14:15|
Finished copying zImage to RAM 
07:04:24/13:14:16|
Found 0 errors checking kernel image
07:04:24/13:14:17|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
07:04:24/13:14:17|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
07:04:24/13:14:17|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
07:04:24/13:14:17|SPIconfig done now to set up the LMK3200 registers
07:04:24/13:14:17|LMK #0 : regInit[0]=0x80000000
07:04:24/13:14:18|LMK #0 : regInit[1]=0x10070600
07:04:24/13:14:18|LMK #0 : regInit[2]=0x60601
07:04:24/13:14:18|LMK #0 : regInit[3]=0x60602
07:04:24/13:14:18|LMK #0 : regInit[4]=0x60603
07:04:24/13:14:18|LMK #0 : regInit[5]=0x70624
07:04:24/13:14:18|LMK #0 : regInit[6]=0x70605
07:04:24/13:14:18|LMK #0 : regInit[7]=0x70606
07:04:24/13:14:18|LMK #0 : regInit[8]=0x70627
07:04:24/13:14:18|LMK #0 : regInit[9]=0x10000908
07:04:24/13:14:18|LMK #0 : regInit[10]=0xA0022A09
07:04:24/13:14:18|LMK #0 : regInit[11]=0x82800B
07:04:24/13:14:18|LMK #0 : regInit[12]=0x28C800D
07:04:24/13:14:18|LMK #0 : regInit[13]=0x830020E
07:04:24/13:14:18|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
07:04:24/13:14:18|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
07:04:24/13:14:18|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
07:04:24/13:14:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
07:04:24/13:14:19|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
07:04:24/13:14:22|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
07:04:24/13:14:28|Using Xilinx Virtex440 machine description
07:04:24/13:14:29|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
07:04:24/13:14:29|Zone PFN ranges:
07:04:24/13:14:29|  DMA      0x00000000 -> 0x00007000
07:04:24/13:14:29|  Normal   0x00007000 -> 0x00007000
07:04:24/13:14:29|Movable zone start PFN for each node
07:04:24/13:14:29|early_node_map[1] active PFN ranges
07:04:24/13:14:29|    0: 0x00000000 -> 0x00007000
07:04:24/13:14:29|MMU: Allocated 1088 bytes of context maps for 255 contexts
07:04:24/13:14:29|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
07:04:24/13:14:29|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
07:04:24/13:14:30|PID hash table entries: 512 (order: 9, 2048 bytes)
07:04:24/13:14:30|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
07:04:24/13:14:30|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
07:04:24/13:14:30|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
07:04:24/13:14:30|Kernel virtual memory layout:
07:04:24/13:14:30|  * 0xffffe000..0xfffff000  : fixmap
07:04:24/13:14:30|  * 0xfde00000..0xfe000000  : consistent mem
07:04:24/13:14:30|  * 0xfde00000..0xfde00000  : early ioremap
07:04:24/13:14:30|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
07:04:24/13:14:30|NR_IRQS:512
07:04:24/13:14:30|clocksource: timebase mult[a00000] shift[22] registered
07:04:24/13:14:30|Console: colour dummy device 80x25
07:04:24/13:14:30|Mount-cache hash table entries: 512
07:04:24/13:14:30|NET: Registered protocol family 16
07:04:24/13:14:30|PCI: Probing PCI hardware
07:04:24/13:14:30|bio: create slab <bio-0> at 0
07:04:24/13:14:30|NET: Registered protocol family 2
07:04:24/13:14:30|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
07:04:24/13:14:30|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
07:04:24/13:14:31|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
07:04:24/13:14:31|TCP: Hash tables configured (established 4096 bind 4096)
07:04:24/13:14:31|TCP reno registered
07:04:24/13:14:31|NET: Registered protocol family 1
07:04:24/13:14:31|ROMFS MTD (C) 2007 Red Hat, Inc.
07:04:24/13:14:31|msgmni has been set to 214
07:04:24/13:14:31|io scheduler noop registered
07:04:24/13:14:31|io scheduler anticipatory registered
07:04:24/13:14:31|io scheduler deadline registered
07:04:24/13:14:31|io scheduler cfq registered (default)
07:04:24/13:14:31|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
07:04:24/13:14:31|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
07:04:24/13:14:31|console [ttyS0] enabled
07:04:24/13:14:31|brd: module loaded
07:04:24/13:14:31|loop: module loaded
07:04:24/13:14:31|Device Tree Probing 'ethernet'
07:04:24/13:14:31|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
07:04:24/13:14:31|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
07:04:24/13:14:31|XLlTemac: DCR address: 0x80
07:04:24/13:14:31|XLlTemac: buffer descriptor size: 32768 (0x8000)
07:04:24/13:14:31|XLlTemac: Allocating DMA descriptors with kmalloc
07:04:24/13:14:31|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
07:04:24/13:14:32|XTemac: PHY detected at address 7.
07:04:24/13:14:32|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
07:04:24/13:14:32|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32| Intel/Sharp Extended Query Table at 0x010A
07:04:24/13:14:32|Using buffer write method
07:04:24/13:14:32|cfi_cmdset_0001: Erase suspend on write enabled
07:04:24/13:14:32|cmdlinepart partition parsing not available
07:04:24/13:14:32|RedBoot partition parsing not available
07:04:24/13:14:32|Creating 5 MTD partitions on "fc000000.flash":
07:04:24/13:14:32|0x000000000000-0x000000500000 : "golden_firmware"
07:04:24/13:14:32|0x000000500000-0x000000800000 : "golden_kernel"
07:04:24/13:14:32|0x000000800000-0x000000d00000 : "user_firmware"
07:04:24/13:14:32|0x000000d00000-0x000000fe0000 : "user_kernel"
07:04:24/13:14:32|0x000000fe0000-0x000001000000 : "env_variables"
07:04:24/13:14:33|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
07:04:24/13:14:33|SPI: XIlinx spi: bus number now 32766
07:04:24/13:14:33|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
07:04:24/13:14:33|SPI: XIlinx spi: bus number now 32765
07:04:24/13:14:33|mice: PS/2 mouse device common for all mice
07:04:24/13:14:33|Device Tree Probing 'i2c'
07:04:24/13:14:33| #0 at 0x81600000 mapped to 0xD1030000, irq=22
07:04:24/13:14:33|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
07:04:24/13:14:33|TCP cubic registered
07:04:24/13:14:33|NET: Registered protocol family 17
07:04:24/13:14:33|RPC: Registered udp transport module.
07:04:24/13:14:33|RPC: Registered tcp transport module.
07:04:24/13:14:33|eth0: XLlTemac: Options: 0x3fa
07:04:24/13:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
07:04:24/13:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
07:04:24/13:14:34|eth0: XLlTemac: speed set to 1000Mb/s
07:04:24/13:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
07:04:24/13:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
07:04:24/13:14:36|Sending DHCP requests ., OK
07:04:24/13:14:37|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.6
07:04:24/13:14:37|IP-Config: Complete:
07:04:24/13:14:37|     device=eth0, addr=192.168.11.6, mask=255.255.255.0, gw=255.255.255.255,
07:04:24/13:14:37|     host=aida06, domain=dl.ac.uk, nis-domain=nuclear.physics,
07:04:24/13:14:37|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06
07:04:24/13:14:37|Looking up port of RPC 100003/2 on 192.168.11.99
07:04:24/13:14:37|Looking up port of RPC 100005/1 on 192.168.11.99
07:04:24/13:14:37|VFS: Mounted root (nfs filesystem) on device 0:12.
07:04:24/13:14:38|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
07:04:24/13:14:38|Starting sysinit...
07:04:24/13:14:38|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
07:04:24/13:14:39|		Press 'I' to enter interactive startup.
07:04:24/13:14:39|Setting clock  (utc): Sun Apr  7 12:14:39 BST 2024 [  OK  ]
07:04:24/13:14:39|Building the cache [  OK  ]
07:04:24/13:14:39|Setting hostname aida06:  [  OK  ]
07:04:24/13:14:41|Mounting local filesystems:  [  OK  ]
07:04:24/13:14:42|Enabling /etc/fstab swaps:  [  OK  ]
07:04:24/13:14:45|Finishing sysinit...
INIT: Entering runlevel: 3
07:04:24/13:14:48|Entering non-interactive startup
07:04:24/13:14:48|FATAL: Module ipv6 not found.
07:04:24/13:14:49|Bringing up loopback interface:  [  OK  ]
07:04:24/13:14:52|FATAL: Module ipv6 not found.
07:04:24/13:14:53|Starting system logger: [  OK  ]
07:04:24/13:14:54|Starting kernel logger: [  OK  ]
07:04:24/13:14:54|Starting rpcbind: [  OK  ]
07:04:24/13:14:56|Mounting NFS filesystems:  [  OK  ]
07:04:24/13:14:56|Mounting other filesystems:  [  OK  ]
07:04:24/13:14:56|Starting xinetd: [  OK  ]
07:04:24/13:14:57|Starting midas:  Starting MIDAS Data Acquisition for aida06
07:04:24/13:14:58|xaida: device parameters: base=0x81000000 size=0x200000
07:04:24/13:15:03|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
07:04:24/13:15:03|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
07:04:24/13:15:03|xaida: driver assigned major number 254
07:04:24/13:15:04|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
07:04:24/13:15:09|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
07:04:24/13:15:09|AIDAMEM: aidamem: driver assigned major number 253
07:04:24/13:15:09|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
07:04:24/13:15:16|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
07:04:24/13:15:16|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
07:04:24/13:15:16|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
07:04:24/13:15:16|Computer Name = aida06; Temp Directory = /tmp/tcl361
07:04:24/13:15:21|
07:04:24/13:15:25|AIDA Data Acquisition Program Release 10.0.Jul  6 2022_15:10:57 starting
07:04:24/13:15:25|
07:04:24/13:15:25|Built without pthreads
07:04:24/13:15:25|
07:04:24/13:15:25|Creating NetVars
07:04:24/13:15:25|Output buffer length = 65504; format option = 4; transfer option = 3
07:04:24/13:15:25|EB transfer option = 3
07:04:24/13:15:26|NetVars creaxaida: open:
07:04:24/13:15:26|ted and initialised
07:04:24/13:15:26|StatisticsAIDAMEM: aidamem_open:
07:04:24/13:15:26| thread starting
07:04:24/13:15:26|Data Acquisition task has PID 375
07:04:24/13:15:26|Statistics thread created
07:04:24/13:15:26|Stat/Rate creation thread starting
07:04:24/13:15:26|Stat/Rate creation thread created
07:04:24/13:15:26|Hit/Rate creation thread starting
07:04:24/13:15:26|Hit/Rate creation thread created
07:04:24/13:15:26|AIDA Heartbeat thread starting
07:04:24/13:15:26|Heartbeat thread created
07:04:24/13:15:26|Installing signal handlers
07:04:24/13:15:26|Done
07:04:24/13:15:26|ModuleNum = 0
07:04:24/13:15:26|Aida Initialise complete. AidaExecV10.0: Build Jul  6 2022_15:10:57. HDL version : 03350706 
07:04:24/13:15:26|package limit is not available: can't find package limit
07:04:24/13:15:26|Running with default file descriptor limit
07:04:24/13:15:26|Spectra table initialised
07:04:24/13:15:26|AIDA Data Acquisition now all ready to start
07:04:24/13:15:26|SIGBUS, SIGSEGV and SIGPIPE traps setup
07:04:24/13:15:26|package setuid is not available: can't find package setuid
07:04:24/13:15:27|Running as user 0 group 0
07:04:24/13:15:27|[  OK  ]
07:04:24/13:15:36|
07:04:24/13:15:36|DENX ELDK version 4.2 build 2008-04-01
07:04:24/13:15:36|Linux 2.6.31 on a ppc
07:04:24/13:15:36|
07:04:24/13:15:36|aida06 login: /debug user
  560   Sun Apr 7 12:02:23 2024 TDSunday 7 April
13.00 Cooling water temperature and flow OK - attachment 1

      Test of AIDA 2x MSL type BB18(DS)-1000 24cm x 8cm DSSSDs 'as is' ( to be defined later in this Elog )

      DSSSD bias & leakage current OK - attachment 2
      Ambient temp 24.7 deg C, d.p. 7.1 deg C, RH 32.4%
      Leakage current c. 6.6uA => 4nA/cm2/100um ( very good )

      FEE64 temps OK - attachment 3
      *except* aida02 ASIC temp which is known to be u/s

      All system wide checks OK *except* aida02 and aida04 WR decoder status - attachment 4

      WR timestamps OK - attachment 5

      WR (info code 4 & 5), correlation scaler, PAUSE, RESUME, DISC, ADC data item stats - attachments 6-12

      per FEE64 Rate spectra - attachments 13-15

      per 1.8.W spectra - 20us FSR - attachments 16-1710129


14.45 FEE64 config check - per https://elog.ph.ed.ac.uk/DESPEC/562


FEE64 # PSU cable #           MAC
1       1                     ?
2       2                     ?
3       3                     ?
4       4                     41 ee 71
5       5                     ?
6       6                     ?
7       7                     f6 5a
8       8                     41 d7 cd
9       ?                     ?
10      10                    41 d0 0e
11      11                    41 ee 0f
12      ?                     ?
13      13                    ?
14      ?                     0d 15
15      15                    ?
16      16                    f6 ed

? = no line of sight

- AIDA FEE64 PSU cabling - see attachment 18

Currently

PSU #1 1-3, 2-4, 9-5, 15-12
PSU #2 14-7, 6-8, 10-13, 11-16

Should be changed to

PSU #1 1-3, 2-4, 9-15, 5-12
PSU #2 14-7, 6-8, 10-11, 13-16


- AIDA snout mount to support frame - attachments 19 & 21
LHS ( looking downstream ) incorrect, RHS OK

- LKs
LK1 fitted aida02, aida04, aida06, aida08
LK3 fitted aida03, aida07, aida01, aida14


Significant gaps in snout ( i.e. possible light leaks ) where bPlas cabling exits snout - attachments 20 & 22

bPlas driver PCBs removed 
bPlas ribbon cables, drain wires and Lemo 00.250 cabling disconnected - some cabling/drain wires touching snout etc


15.15 Adjust bPlas ribbon cables/drain wires/Lemo cables to ensure that none are touching snout or AIDA support frame - significant improvement

      DSSSD bias & leakage current OK - attachment 23

      FEE64 temps OK - attachment 24

      ADC data item stats - attachments 25

      per FEE64 Rate spectra - attachments 26
       6x FEE64s < 20k. all FEE64s < 120k ( cf. 4x FEE64s < 20k, 5x FEE64s > 100k, 1x FEE64 320k earlier - attachment 12 )

      per 1.8.W spectra - 20us FSR - attachments 27-28

17.15

LHS top hat washers
LK1, LK3
PSU power cables
tighten gnd screws
attachments 29-34

Note that some of the newly installed FEE64s have the incorrect rails fitted - FEE64s may ground to AIDA support stand

17.55 CAEN N1419ET LK removed -> floating outputs

18.15 +LK1 restored
attachments 35-40

18.40 +pwwer cycle
attachments 41-46
Attachment 1: 20240407_125941.jpg
20240407_125941.jpg
Attachment 2: Screenshot_from_2024-04-07_13-20-57.png
Screenshot_from_2024-04-07_13-20-57.png
Attachment 3: Screenshot_from_2024-04-07_13-29-54.png
Screenshot_from_2024-04-07_13-29-54.png
Attachment 4: Screenshot_from_2024-04-07_13-32-14.png
Screenshot_from_2024-04-07_13-32-14.png
Attachment 5: Screenshot_from_2024-04-07_13-36-47.png
Screenshot_from_2024-04-07_13-36-47.png
Attachment 6: Screenshot_from_2024-04-07_13-42-52.png
Screenshot_from_2024-04-07_13-42-52.png
Attachment 7: Screenshot_from_2024-04-07_13-42-26.png
Screenshot_from_2024-04-07_13-42-26.png
Attachment 8: Screenshot_from_2024-04-07_13-42-04.png
Screenshot_from_2024-04-07_13-42-04.png
Attachment 9: Screenshot_from_2024-04-07_13-41-37.png
Screenshot_from_2024-04-07_13-41-37.png
Attachment 10: Screenshot_from_2024-04-07_13-41-25.png
Screenshot_from_2024-04-07_13-41-25.png
Attachment 11: Screenshot_from_2024-04-07_13-41-03.png
Screenshot_from_2024-04-07_13-41-03.png
Attachment 12: Screenshot_from_2024-04-07_13-40-35.png
Screenshot_from_2024-04-07_13-40-35.png
Attachment 13: Screenshot_from_2024-04-07_13-47-03.png
Screenshot_from_2024-04-07_13-47-03.png
Attachment 14: Screenshot_from_2024-04-07_13-46-47.png
Screenshot_from_2024-04-07_13-46-47.png
Attachment 15: Screenshot_from_2024-04-07_13-46-39.png
Screenshot_from_2024-04-07_13-46-39.png
Attachment 16: Screenshot_from_2024-04-07_13-53-21.png
Screenshot_from_2024-04-07_13-53-21.png
Attachment 17: Screenshot_from_2024-04-07_13-52-25.png
Screenshot_from_2024-04-07_13-52-25.png
Attachment 18: 20240407_143456.jpg
20240407_143456.jpg
Attachment 19: 20240407_143510.jpg
20240407_143510.jpg
Attachment 20: 20240407_143553.jpg
20240407_143553.jpg
Attachment 21: 20240407_143532.jpg
20240407_143532.jpg
Attachment 22: 20240407_143539.jpg
20240407_143539.jpg
Attachment 23: Screenshot_from_2024-04-07_15-10-39.png
Screenshot_from_2024-04-07_15-10-39.png
Attachment 24: Screenshot_from_2024-04-07_15-10-53.png
Screenshot_from_2024-04-07_15-10-53.png
Attachment 25: Screenshot_from_2024-04-07_15-13-00.png
Screenshot_from_2024-04-07_15-13-00.png
Attachment 26: Screenshot_from_2024-04-07_15-12-49.png
Screenshot_from_2024-04-07_15-12-49.png
Attachment 27: Screenshot_from_2024-04-07_15-11-54.png
Screenshot_from_2024-04-07_15-11-54.png
Attachment 28: Screenshot_from_2024-04-07_15-11-06.png
Screenshot_from_2024-04-07_15-11-06.png
Attachment 29: Screenshot_from_2024-04-07_17-14-23.png
Screenshot_from_2024-04-07_17-14-23.png
Attachment 30: Screenshot_from_2024-04-07_17-13-49.png
Screenshot_from_2024-04-07_17-13-49.png
Attachment 31: Screenshot_from_2024-04-07_17-11-59.png
Screenshot_from_2024-04-07_17-11-59.png
Attachment 32: Screenshot_from_2024-04-07_17-11-47.png
Screenshot_from_2024-04-07_17-11-47.png
Attachment 33: Screenshot_from_2024-04-07_17-11-38.png
Screenshot_from_2024-04-07_17-11-38.png
Attachment 34: Screenshot_from_2024-04-07_17-11-19.png
Screenshot_from_2024-04-07_17-11-19.png
Attachment 35: Screenshot_from_2024-04-07_18-12-38.png
Screenshot_from_2024-04-07_18-12-38.png
Attachment 36: Screenshot_from_2024-04-07_18-12-53.png
Screenshot_from_2024-04-07_18-12-53.png
Attachment 37: Screenshot_from_2024-04-07_18-13-03.png
Screenshot_from_2024-04-07_18-13-03.png
Attachment 38: Screenshot_from_2024-04-07_18-14-51.png
Screenshot_from_2024-04-07_18-14-51.png
Attachment 39: Screenshot_from_2024-04-07_18-14-12.png
Screenshot_from_2024-04-07_18-14-12.png
Attachment 40: Screenshot_from_2024-04-07_18-13-10.png
Screenshot_from_2024-04-07_18-13-10.png
Attachment 41: Screenshot_from_2024-04-07_18-44-32.png
Screenshot_from_2024-04-07_18-44-32.png
Attachment 42: Screenshot_from_2024-04-07_18-44-08.png
Screenshot_from_2024-04-07_18-44-08.png
Attachment 43: Screenshot_from_2024-04-07_18-43-30.png
Screenshot_from_2024-04-07_18-43-30.png
Attachment 44: Screenshot_from_2024-04-07_18-43-21.png
Screenshot_from_2024-04-07_18-43-21.png
Attachment 45: Screenshot_from_2024-04-07_18-43-12.png
Screenshot_from_2024-04-07_18-43-12.png
Attachment 46: Screenshot_from_2024-04-07_18-42-56.png
Screenshot_from_2024-04-07_18-42-56.png
  559   Wed Apr 3 13:02:19 2024 NHMerger for 16 FEEs
Changed /MIDAS/Linux/startup/NewMerger

Change parameters -i and -l in master64 to 16 for 16 FEEs
Update NewMerger Options LinksAvailable to 16, LinksInUse to 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%
Fix NetVar RunOptions 1 (was 0)

Restart Merger HTTPd, Tape, Merger, MBS Spy

Reset/Setup/Go 

16 Links green and status going, all good?

Bias DSSSDs and turn data transfer on

Merger connected, shows rate and updates... no rate in Tape Server?

.. Oops forget to turn on Output to data storage in merger!

Rate in tape server and to MBS: 7 MB/s

Merger, Tape and MBS working with 16 FEEs
Attachment 1: Screenshot_from_2024-04-03_14-15-02.png
Screenshot_from_2024-04-03_14-15-02.png
Attachment 2: Screenshot_from_2024-04-03_14-15-25.png
Screenshot_from_2024-04-03_14-15-25.png
Attachment 3: Screenshot_from_2024-04-03_14-15-36.png
Screenshot_from_2024-04-03_14-15-36.png
Attachment 4: Screenshot_from_2024-04-03_14-15-45.png
Screenshot_from_2024-04-03_14-15-45.png
Attachment 5: Screenshot_from_2024-04-03_14-15-52.png
Screenshot_from_2024-04-03_14-15-52.png
  558   Wed Apr 3 12:42:57 2024 NHReport aida02 WR errors

The WR error counter for aida02 seems to consantly rise

Tried reseating cable on both ends, no change

However clock status passed, aida02 has a correct WR timestamp and no FIFO/PLL errors seen

Edit to add: aida02 has the faulty ASIC temperature readout as well, related or coincidence?

Attachment 1: Screenshot_from_2024-04-03_13-43-48.png
Screenshot_from_2024-04-03_13-43-48.png
Attachment 2: Screenshot_from_2024-04-03_13-43-38.png
Screenshot_from_2024-04-03_13-43-38.png
Attachment 3: Screenshot_from_2024-04-03_13-43-58.png
Screenshot_from_2024-04-03_13-43-58.png
Attachment 4: Screenshot_from_2024-04-03_13-44-14.png
Screenshot_from_2024-04-03_13-44-14.png
Attachment 5: Screenshot_from_2024-04-03_13-44-36.png
Screenshot_from_2024-04-03_13-44-36.png
  557   Wed Apr 3 12:09:47 2024 NHReport - aida06 frequently fails to boot first time (PHY error)

When booting up AIDA aida06 usually crashes the first time, it fails to get IP from DHCP
After 180 seconds it reboots and seems to connect fine

Log file attached, key part (to me) is this:

27:03:24/14:13:19|Sending DHCP requests .
27:03:24/14:13:21|eth0: XLlTemac: PHY Link carrier lost.
27:03:24/14:13:21|..... timed out!
27:03:24/14:14:33|IP-Config: Reopening network devices...
27:03:24/14:14:33|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:14:34|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:14:36|Sending DHCP requests ......
27:03:24/14:26:16|
Attachment 1: ttyUSB15
27:03:24/14:12:59|
ISOL Version 1.00 Date 9th January 2017
27:03:24/14:12:59|
Flash base address=FC000000
27:03:24/14:12:59|
Set Flash to ASync Mode
27:03:24/14:12:59|
XST_SUCCESS
27:03:24/14:12:59|
Finished copying zImage to RAM 
27:03:24/14:12:59|
27:03:24/14:12:59|
Found 0 errors checking kernel image
27:03:24/14:13:00|VHDL version number 0X03350706 
27:03:24/14:13:00|
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
27:03:24/14:13:00|
Starting LMK 3200 setup 
27:03:24/14:13:00|
27:03:24/14:13:00|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
27:03:24/14:13:00|
.... SPI Base Address=0x81400000
27:03:24/14:13:00|
clk_control_reg=0x4
27:03:24/14:13:01|Next step is SPIconfig
27:03:24/14:13:01|
Control 32(0x81400000)=0x180
27:03:24/14:13:01|
SlaveSel(0x81400000)=0x3
27:03:24/14:13:01|
Ctrl(0x81400000)=0xE6
27:03:24/14:13:01|
Ctrl(0x81400000)=0x86
27:03:24/14:13:01|SPIconfig done now to set up the LMK3200 registers
27:03:24/14:13:01|LMK #0 : regInit[0]=0x80000000
27:03:24/14:13:01|LMK #0 : regInit[1]=0x10070600
27:03:24/14:13:01|LMK #0 : regInit[2]=0x60601
27:03:24/14:13:01|LMK #0 : regInit[3]=0x60602
27:03:24/14:13:01|LMK #0 : regInit[4]=0x60603
27:03:24/14:13:01|LMK #0 : regInit[5]=0x70624
27:03:24/14:13:01|LMK #0 : regInit[6]=0x70605
27:03:24/14:13:01|LMK #0 : regInit[7]=0x70606
27:03:24/14:13:01|LMK #0 : regInit[8]=0x70627
27:03:24/14:13:01|LMK #0 : regInit[9]=0x10000908
27:03:24/14:13:01|LMK #0 : regInit[10]=0xA0022A09
27:03:24/14:13:01|LMK #0 : regInit[11]=0x82800B
27:03:24/14:13:01|LMK #0 : regInit[12]=0x28C800D
27:03:24/14:13:01|LMK #0 : regInit[13]=0x830020E
27:03:24/14:13:01|LMK #0 : regInit[14]=0xC800180F
27:03:24/14:13:01|
Calibrate completed at 943 counts
27:03:24/14:13:01|
Setting Clock Control =0x0000000B, to set GOE and sync bit
27:03:24/14:13:01|
Ctrl @ SPIstop (0x81400000)=0x186
27:03:24/14:13:01|
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
27:03:24/14:13:01|
27:03:24/14:13:01|
Finished Clock setup LMK03200
27:03:24/14:13:01|
completed LMK 3200 setup 
27:03:24/14:13:02|
Loaded all four ASICs with default settings 
27:03:24/14:13:02|
Setting the ADCs into calibration mode 
27:03:24/14:13:02|
27:03:24/14:13:02|
Control 32(0x81400400)=0x180
27:03:24/14:13:02|
SlaveSel(0x81400400)=0xFF
27:03:24/14:13:02|
Ctrl(0x81400400)=0xE6
27:03:24/14:13:02|
Ctrl(0x81400400)=0x86
27:03:24/14:13:02|
Init : Config of AD9252 SPI ok
27:03:24/14:13:02|
27:03:24/14:13:02|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
27:03:24/14:13:02|
Cal not completed 
27:03:24/14:13:02|
ADC calibrate failed 
27:03:24/14:13:02|
Jumping to kernel simpleboot...
27:03:24/14:13:02|
27:03:24/14:13:02|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
27:03:24/14:13:02|
Allocating 0x3b78cc bytes for kernel ...
27:03:24/14:13:02|
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
27:03:24/14:13:05|
27:03:24/14:13:05|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
27:03:24/14:13:12|
Finalizing device tree... flat tree at 0xbd1300
27:03:24/14:13:12|
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
27:03:24/14:13:12|Using Xilinx Virtex440 machine description
27:03:24/14:13:12|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
27:03:24/14:13:12|Zone PFN ranges:
27:03:24/14:13:12|  DMA      0x00000000 -> 0x00007000
27:03:24/14:13:12|  Normal   0x00007000 -> 0x00007000
27:03:24/14:13:12|Movable zone start PFN for each node
27:03:24/14:13:13|early_node_map[1] active PFN ranges
27:03:24/14:13:13|    0: 0x00000000 -> 0x00007000
27:03:24/14:13:13|MMU: Allocated 1088 bytes of context maps for 255 contexts
27:03:24/14:13:13|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
27:03:24/14:13:13|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
27:03:24/14:13:13|PID hash table entries: 512 (order: 9, 2048 bytes)
27:03:24/14:13:13|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
27:03:24/14:13:13|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
27:03:24/14:13:13|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
27:03:24/14:13:13|Kernel virtual memory layout:
27:03:24/14:13:13|  * 0xffffe000..0xfffff000  : fixmap
27:03:24/14:13:13|  * 0xfde00000..0xfe000000  : consistent mem
27:03:24/14:13:13|  * 0xfde00000..0xfde00000  : early ioremap
27:03:24/14:13:13|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
27:03:24/14:13:13|NR_IRQS:512
27:03:24/14:13:13|clocksource: timebase mult[a00000] shift[22] registered
27:03:24/14:13:13|Console: colour dummy device 80x25
27:03:24/14:13:13|Mount-cache hash table entries: 512
27:03:24/14:13:13|NET: Registered protocol family 16
27:03:24/14:13:14|PCI: Probing PCI hardware
27:03:24/14:13:14|bio: create slab <bio-0> at 0
27:03:24/14:13:14|NET: Registered protocol family 2
27:03:24/14:13:14|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
27:03:24/14:13:14|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
27:03:24/14:13:14|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
27:03:24/14:13:14|TCP: Hash tables configured (established 4096 bind 4096)
27:03:24/14:13:14|TCP reno registered
27:03:24/14:13:14|NET: Registered protocol family 1
27:03:24/14:13:14|ROMFS MTD (C) 2007 Red Hat, Inc.
27:03:24/14:13:14|msgmni has been set to 214
27:03:24/14:13:14|io scheduler noop registered
27:03:24/14:13:14|io scheduler anticipatory registered
27:03:24/14:13:14|io scheduler deadline registered
27:03:24/14:13:14|io scheduler cfq registered (default)
27:03:24/14:13:14|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
27:03:24/14:13:14|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
27:03:24/14:13:14|console [ttyS0] enabled
27:03:24/14:13:14|brd: module loaded
27:03:24/14:13:14|loop: module loaded
27:03:24/14:13:14|Device Tree Probing 'ethernet'
27:03:24/14:13:14|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
27:03:24/14:13:14|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
27:03:24/14:13:15|XLlTemac: DCR address: 0x80
27:03:24/14:13:15|XLlTemac: buffer descriptor size: 32768 (0x8000)
27:03:24/14:13:15|XLlTemac: Allocating DMA descriptors with kmalloc
27:03:24/14:13:15|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
27:03:24/14:13:15|XTemac: PHY detected at address 7.
27:03:24/14:13:15|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
27:03:24/14:13:15|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:13:15|Using buffer write method
27:03:24/14:13:15|cfi_cmdset_0001: Erase suspend on write enabled
27:03:24/14:13:15|cmdlinepart partition parsing not available
27:03:24/14:13:15|RedBoot partition parsing not available
27:03:24/14:13:15|Creating 5 MTD partitions on "fc000000.flash":
27:03:24/14:13:15|0x000000000000-0x000000500000 : "golden_firmware"
27:03:24/14:13:16|0x000000500000-0x000000800000 : "golden_kernel"
27:03:24/14:13:16|0x000000800000-0x000000d00000 : "user_firmware"
27:03:24/14:13:16|0x000000d00000-0x000000fe0000 : "user_kernel"
27:03:24/14:13:16|0x000000fe0000-0x000001000000 : "env_variables"
27:03:24/14:13:16|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
27:03:24/14:13:16|SPI: XIlinx spi: bus number now 32766
27:03:24/14:13:16|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
27:03:24/14:13:16|SPI: XIlinx spi: bus number now 32765
27:03:24/14:13:16|mice: PS/2 mouse device common for all mice
27:03:24/14:13:16|Device Tree Probing 'i2c'
27:03:24/14:13:16| #0 at 0x81600000 mapped to 0xD1030000, irq=22
27:03:24/14:13:16|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
27:03:24/14:13:16|TCP cubic registered
27:03:24/14:13:16|NET: Registered protocol family 17
27:03:24/14:13:16|RPC: Registered udp transport module.
27:03:24/14:13:16|RPC: Registered tcp transport module.
27:03:24/14:13:16|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:13:17|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:13:17|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:13:17|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:13:19|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:13:19|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:13:19|Sending DHCP requests .
27:03:24/14:13:21|eth0: XLlTemac: PHY Link carrier lost.
27:03:24/14:13:21|..... timed out!
27:03:24/14:14:33|IP-Config: Reopening network devices...
27:03:24/14:14:33|eth0: XLlTemac: Options: 0x3fa
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
27:03:24/14:14:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
27:03:24/14:14:34|eth0: XLlTemac: speed set to 1000Mb/s
27:03:24/14:14:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
27:03:24/14:14:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
27:03:24/14:14:36|Sending DHCP requests ......
27:03:24/14:26:16|
ISOL Version 1.00 Date 9th January 2017
27:03:24/14:26:16|
Flash base address=FC000000
27:03:24/14:26:16|
Set Flash to ASync Mode
27:03:24/14:26:16|
XST_SUCCESS
27:03:24/14:26:16|
Finished copying zImage to RAM 
27:03:24/14:26:17|
27:03:24/14:26:17|
Found 0 errors checking kernel image
27:03:24/14:26:18|VHDL version number 0X03350706 
27:03:24/14:26:18|
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
27:03:24/14:26:18|
Starting LMK 3200 setup 
27:03:24/14:26:18|
27:03:24/14:26:18|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
27:03:24/14:26:18|
.... SPI Base Address=0x81400000
27:03:24/14:26:18|
clk_control_reg=0x4
27:03:24/14:26:18|Next step is SPIconfig
27:03:24/14:26:18|
Control 32(0x81400000)=0x180
27:03:24/14:26:18|
SlaveSel(0x81400000)=0x3
27:03:24/14:26:18|
Ctrl(0x81400000)=0xE6
27:03:24/14:26:18|
Ctrl(0x81400000)=0x86
27:03:24/14:26:18|SPIconfig done now to set up the LMK3200 registers
27:03:24/14:26:18|LMK #0 : regInit[0]=0x80000000
27:03:24/14:26:18|LMK #0 : regInit[1]=0x10070600
27:03:24/14:26:18|LMK #0 : regInit[2]=0x60601
27:03:24/14:26:19|LMK #0 : regInit[3]=0x60602
27:03:24/14:26:19|LMK #0 : regInit[4]=0x60603
27:03:24/14:26:19|LMK #0 : regInit[5]=0x70624
27:03:24/14:26:19|LMK #0 : regInit[6]=0x70605
27:03:24/14:26:19|LMK #0 : regInit[7]=0x70606
27:03:24/14:26:19|LMK #0 : regInit[8]=0x70627
27:03:24/14:26:19|LMK #0 : regInit[9]=0x10000908
27:03:24/14:26:19|LMK #0 : regInit[10]=0xA0022A09
27:03:24/14:26:19|LMK #0 : regInit[11]=0x82800B
27:03:24/14:26:19|LMK #0 : regInit[12]=0x28C800D
27:03:24/14:26:19|LMK #0 : regInit[13]=0x830020E
27:03:24/14:26:19|LMK #0 : regInit[14]=0xC800180F
27:03:24/14:26:19|
Calibrate completed at 943 counts
27:03:24/14:26:19|
Setting Clock Control =0x0000000B, to set GOE and sync bit
27:03:24/14:26:19|
Ctrl @ SPIstop (0x81400000)=0x186
27:03:24/14:26:19|
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
27:03:24/14:26:19|
27:03:24/14:26:19|
Finished Clock setup LMK03200
27:03:24/14:26:19|
completed LMK 3200 setup 
27:03:24/14:26:19|
Loaded all four ASICs with default settings 
27:03:24/14:26:19|
Setting the ADCs into calibration mode 
27:03:24/14:26:19|
27:03:24/14:26:19|
Control 32(0x81400400)=0x180
27:03:24/14:26:19|
SlaveSel(0x81400400)=0xFF
27:03:24/14:26:19|
Ctrl(0x81400400)=0xE6
27:03:24/14:26:19|
Ctrl(0x81400400)=0x86
27:03:24/14:26:19|
Init : Config of AD9252 SPI ok
27:03:24/14:26:19|
27:03:24/14:26:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
27:03:24/14:26:20|
ADCs calibrated
27:03:24/14:26:20|
27:03:24/14:26:20|
Control 32(0x81400400)=0x186
27:03:24/14:26:20|
SlaveSel(0x81400400)=0xFF
27:03:24/14:26:20|
Ctrl(0x81400400)=0xE6
27:03:24/14:26:20|
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
27:03:24/14:26:20|
27:03:24/14:26:20|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
27:03:24/14:26:20|
27:03:24/14:26:20|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
27:03:24/14:26:20|
Allocating 0x3b78cc bytes for kernel ...
27:03:24/14:26:20|
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
27:03:24/14:26:23|
27:03:24/14:26:23|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
27:03:24/14:26:29|
Finalizing device tree... flat tree at 0xbd1300
27:03:24/14:26:29|
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x71 
27:03:24/14:26:30|Using Xilinx Virtex440 machine description
27:03:24/14:26:30|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
27:03:24/14:26:30|Zone PFN ranges:
27:03:24/14:26:30|  DMA      0x00000000 -> 0x00007000
27:03:24/14:26:30|  Normal   0x00007000 -> 0x00007000
27:03:24/14:26:30|Movable zone start PFN for each node
27:03:24/14:26:30|early_node_map[1] active PFN ranges
27:03:24/14:26:30|    0: 0x00000000 -> 0x00007000
27:03:24/14:26:30|MMU: Allocated 1088 bytes of context maps for 255 contexts
27:03:24/14:26:30|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
27:03:24/14:26:31|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
27:03:24/14:26:31|PID hash table entries: 512 (order: 9, 2048 bytes)
27:03:24/14:26:31|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
27:03:24/14:26:31|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
27:03:24/14:26:31|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
27:03:24/14:26:31|Kernel virtual memory layout:
27:03:24/14:26:31|  * 0xffffe000..0xfffff000  : fixmap
27:03:24/14:26:31|  * 0xfde00000..0xfe000000  : consistent mem
27:03:24/14:26:31|  * 0xfde00000..0xfde00000  : early ioremap
27:03:24/14:26:31|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
27:03:24/14:26:31|NR_IRQS:512
27:03:24/14:26:31|clocksource: timebase mult[a00000] shift[22] registered
27:03:24/14:26:31|Console: colour dummy device 80x25
27:03:24/14:26:31|Mount-cache hash table entries: 512
27:03:24/14:26:31|NET: Registered protocol family 16
27:03:24/14:26:31|PCI: Probing PCI hardware
27:03:24/14:26:31|bio: create slab <bio-0> at 0
27:03:24/14:26:31|NET: Registered protocol family 2
27:03:24/14:26:31|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
27:03:24/14:26:31|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
27:03:24/14:26:32|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
27:03:24/14:26:32|TCP: Hash tables configured (established 4096 bind 4096)
27:03:24/14:26:32|TCP reno registered
27:03:24/14:26:32|NET: Registered protocol family 1
27:03:24/14:26:32|ROMFS MTD (C) 2007 Red Hat, Inc.
27:03:24/14:26:32|msgmni has been set to 214
27:03:24/14:26:32|io scheduler noop registered
27:03:24/14:26:32|io scheduler anticipatory registered
27:03:24/14:26:32|io scheduler deadline registered
27:03:24/14:26:32|io scheduler cfq registered (default)
27:03:24/14:26:32|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
27:03:24/14:26:32|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
27:03:24/14:26:32|console [ttyS0] enabled
27:03:24/14:26:32|brd: module loaded
27:03:24/14:26:32|loop: module loaded
27:03:24/14:26:32|Device Tree Probing 'ethernet'
27:03:24/14:26:32|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:71
27:03:24/14:26:32|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
27:03:24/14:26:32|XLlTemac: DCR address: 0x80
27:03:24/14:26:32|XLlTemac: buffer descriptor size: 32768 (0x8000)
27:03:24/14:26:32|XLlTemac: Allocating DMA descriptors with kmalloc
27:03:24/14:26:32|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
27:03:24/14:26:33|XTemac: PHY detected at address 7.
27:03:24/14:26:33|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
27:03:24/14:26:33|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33| Intel/Sharp Extended Query Table at 0x010A
27:03:24/14:26:33|Using buffer write method
27:03:24/14:26:33|cfi_cmdset_0001: Erase suspend on write enabled
27:03:24/14:26:33|cmdlinepart partition parsing not available
27:03:24/14:26:33|RedBoot partition parsing not available
27:03:24/14:26:33|Creating 5 MTD partitions on "fc000000.flash":
27:03:24/14:26:33|0x000000000000-0x000000500000 : "golden_firmware"
27:03:24/14:26:33|0x000000500000-0x000000800000 : "golden_kernel"
... 1850 more lines ...
  556   Tue Apr 2 18:37:21 2024 TDS505 ADC offsets
S505 ADC offsets using pulser walkthrough data from data file R1

ch = channel + ( module * 64 ) + ( range * 2048 )

adc_data( ch ) = INT( RSHIFT( ABS( adc_data( ch ) - 32768 ), 3 ) - offset( ch ) + 0.5 )
Attachment 1: S505_calibration_data.txt
  $variables
  offset(   0) =    28.32
  offset(   1) =    17.71
  offset(   2) =    -9.75
  offset(   3) =    -5.24
  offset(   4) =   -10.84
  offset(   5) =     0.13
  offset(   6) =    28.59
  offset(   7) =    28.54
  offset(   8) =     0.83
  offset(   9) =   -12.15
  offset(  10) =    29.36
  offset(  11) =   -50.47
  offset(  12) =    10.03
  offset(  13) =   -26.90
  offset(  14) =    12.49
  offset(  15) =    10.57
  offset(  16) =     9.29
  offset(  17) =     1.09
  offset(  18) =    21.86
  offset(  19) =    -0.32
  offset(  20) =    22.59
  offset(  21) = -9999.99
  offset(  22) =   -20.22
  offset(  23) =     6.61
  offset(  24) =    -8.80
  offset(  25) =    -1.15
  offset(  26) =    13.08
  offset(  27) =    -9.03
  offset(  28) =    -2.67
  offset(  29) =    -9.42
  offset(  30) =     8.07
  offset(  31) =    19.01
  offset(  32) =     6.86
  offset(  33) =    15.16
  offset(  34) =    -3.89
  offset(  35) =   -18.80
  offset(  36) =    17.79
  offset(  37) =    16.45
  offset(  38) =    -0.74
  offset(  39) =     0.69
  offset(  40) =   -14.59
  offset(  41) =   -10.02
  offset(  42) =    11.38
  offset(  43) =    -2.57
  offset(  44) =    15.48
  offset(  45) =    14.57
  offset(  46) =     2.95
  offset(  47) =    30.70
  offset(  48) =   -12.55
  offset(  49) =   -15.61
  offset(  50) =    -6.61
  offset(  51) =    -7.10
  offset(  52) =   -33.20
  offset(  53) =    21.42
  offset(  54) =    -4.07
  offset(  55) =   -18.23
  offset(  56) =     8.83
  offset(  57) =   -22.74
  offset(  58) =    11.47
  offset(  59) =    10.67
  offset(  60) =   -28.65
  offset(  61) =    15.29
  offset(  62) =     5.76
  offset(  63) =   -11.57
  offset(  64) =     1.95
  offset(  65) = -9999.99
  offset(  66) =    22.52
  offset(  67) =    42.10
  offset(  68) =    -0.20
  offset(  69) =     9.78
  offset(  70) =   -11.48
  offset(  71) =    -0.03
  offset(  72) =   -12.50
  offset(  73) =    25.08
  offset(  74) =    12.77
  offset(  75) =    -2.76
  offset(  76) =     7.90
  offset(  77) =     9.85
  offset(  78) =     5.20
  offset(  79) =   -13.91
  offset(  80) =    18.82
  offset(  81) =    -8.25
  offset(  82) =    -8.40
  offset(  83) =    13.67
  offset(  84) =   -10.37
  offset(  85) =    28.32
  offset(  86) =    11.42
  offset(  87) =     2.74
  offset(  88) =   -24.88
  offset(  89) =    11.28
  offset(  90) =   -26.69
  offset(  91) =     0.91
  offset(  92) =     8.04
  offset(  93) =     7.64
  offset(  94) =    10.10
  offset(  95) =   -20.71
  offset(  96) =     6.72
  offset(  97) =   -12.63
  offset(  98) =    10.27
  offset(  99) =    16.05
  offset( 100) =     8.96
  offset( 101) =    33.57
  offset( 102) =    28.81
  offset( 103) =    23.49
  offset( 104) =     7.96
  offset( 105) =     6.07
  offset( 106) =    21.57
  offset( 107) =    13.06
  offset( 108) =     0.82
  offset( 109) =    -3.55
  offset( 110) =    -5.21
  offset( 111) =    11.48
  offset( 112) =    16.21
  offset( 113) =    -0.10
  offset( 114) =     3.87
  offset( 115) =    19.76
  offset( 116) =    -9.11
  offset( 117) =    24.64
  offset( 118) =     1.82
  offset( 119) = -9999.99
  offset( 120) =    22.80
  offset( 121) =    -7.37
  offset( 122) =   -18.32
  offset( 123) =   -24.74
  offset( 124) =    -4.71
  offset( 125) =    19.64
  offset( 126) =    53.61
  offset( 127) =     6.57
  offset( 128) =     3.21
  offset( 129) =     7.20
  offset( 130) =     5.55
  offset( 131) =    -3.40
  offset( 132) =   -16.40
  offset( 133) =    -1.71
  offset( 134) =   -14.85
  offset( 135) =   -29.53
  offset( 136) =     0.68
  offset( 137) =    -0.41
  offset( 138) =    17.59
  offset( 139) =   -28.92
  offset( 140) =    -8.79
  offset( 141) =    11.69
  offset( 142) =    -5.80
  offset( 143) =    -8.21
  offset( 144) =    13.90
  offset( 145) =    11.27
  offset( 146) =   -17.02
  offset( 147) =    -8.19
  offset( 148) =     6.14
  offset( 149) =   -13.13
  offset( 150) =    15.41
  offset( 151) =     0.12
  offset( 152) =    -5.02
  offset( 153) =     4.25
  offset( 154) =     3.84
  offset( 155) =    13.34
  offset( 156) =    16.79
  offset( 157) =   -36.21
  offset( 158) =    -2.18
  offset( 159) =    12.94
  offset( 160) =     2.05
  offset( 161) =   -17.31
  offset( 162) =    18.38
  offset( 163) =    -7.17
  offset( 164) =     6.59
  offset( 165) =   -13.36
  offset( 166) =   -12.11
  offset( 167) =    -3.91
  offset( 168) =    10.97
  offset( 169) =   -25.72
  offset( 170) =   -11.53
  offset( 171) =     1.94
  offset( 172) =   -12.76
  offset( 173) =    18.55
  offset( 174) =    13.49
  offset( 175) =    -9.45
  offset( 176) =   -10.19
  offset( 177) =    18.69
  offset( 178) =    24.49
  offset( 179) =    30.44
  offset( 180) =    41.95
  offset( 181) =     1.55
  offset( 182) =    20.88
  offset( 183) =     4.76
  offset( 184) =    15.80
  offset( 185) =    12.37
  offset( 186) =   -13.03
  offset( 187) =    15.64
  offset( 188) =    -8.44
  offset( 189) =    -3.02
  offset( 190) =    10.57
  offset( 191) =     6.88
  offset( 192) =   -17.81
  offset( 193) =    -5.77
  offset( 194) =   -18.29
  offset( 195) =    17.62
  offset( 196) =   -30.66
  offset( 197) =    -7.37
  offset( 198) =    36.32
  offset( 199) =     3.42
  offset( 200) =   -21.92
  offset( 201) =    15.88
  offset( 202) =   -17.33
  offset( 203) =    -2.71
  offset( 204) =   -21.06
  offset( 205) =    -4.53
  offset( 206) =    -1.59
  offset( 207) =    -9.54
  offset( 208) =    -0.04
  offset( 209) =    23.57
  offset( 210) =    13.83
  offset( 211) =    -9.28
  offset( 212) =    12.42
  offset( 213) =    23.85
  offset( 214) =     7.90
  offset( 215) =    -7.29
  offset( 216) =    26.18
  offset( 217) =    33.05
  offset( 218) =    32.10
  offset( 219) =    -8.10
  offset( 220) =    16.16
  offset( 221) =     3.94
  offset( 222) =   -18.22
  offset( 223) =   -16.84
  offset( 224) =     5.79
  offset( 225) =    21.07
  offset( 226) =    45.18
  offset( 227) =   -13.85
  offset( 228) =     6.76
  offset( 229) =    13.59
  offset( 230) =     8.28
  offset( 231) =    -9.32
  offset( 232) =    21.30
  offset( 233) =     7.02
  offset( 234) =    24.11
  offset( 235) =    16.26
  offset( 236) =    11.46
  offset( 237) =     2.16
  offset( 238) =     7.19
  offset( 239) =    18.45
  offset( 240) =     6.65
  offset( 241) =     6.56
  offset( 242) =   -33.59
  offset( 243) =    14.71
  offset( 244) =    21.28
  offset( 245) =    -0.73
  offset( 246) =   -10.29
  offset( 247) =    28.50
  offset( 248) =   -11.37
  offset( 249) =   -18.92
  offset( 250) =    23.12
  offset( 251) =     8.87
  offset( 252) =     9.64
  offset( 253) =    16.43
  offset( 254) =    -2.70
  offset( 255) =    26.14
  offset( 256) =    -4.99
  offset( 257) =   -10.88
  offset( 258) =    27.41
  offset( 259) =    -3.48
  offset( 260) =   -11.18
  offset( 261) =    12.60
  offset( 262) =    22.19
  offset( 263) =   -20.16
  offset( 264) =    20.32
  offset( 265) =     9.66
  offset( 266) =    13.30
  offset( 267) =   -12.64
  offset( 268) =    53.69
  offset( 269) =    23.15
  offset( 270) =    43.60
  offset( 271) =    48.58
  offset( 272) =   -13.12
  offset( 273) =   -12.57
  offset( 274) =    35.19
  offset( 275) =    14.84
  offset( 276) =    33.05
  offset( 277) =    13.73
  offset( 278) =    45.45
  offset( 279) =    55.49
  offset( 280) =    48.13
  offset( 281) =     3.86
  offset( 282) =    28.44
  offset( 283) =    23.73
  offset( 284) =    11.53
  offset( 285) =   -19.46
  offset( 286) =    10.81
  offset( 287) =    17.23
  offset( 288) =   -13.85
  offset( 289) =    30.17
  offset( 290) =    10.70
  offset( 291) =    28.06
  offset( 292) =   -14.35
  offset( 293) =    23.46
  offset( 294) =    15.16
  offset( 295) =   -25.15
  offset( 296) =   -15.03
  offset( 297) =   -24.74
  offset( 298) =    -5.94
... 215 more lines ...
  555   Tue Apr 2 12:36:25 2024 JB, CC, NHInstalling FEE64s of DSSSD2 cont.

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly

 

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

 

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

 

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

 

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

 

- check test  and test - cable daisy chains are removed

 

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

 

  553   Thu Mar 28 09:18:53 2024 TDInstalling FEE64s of DSSSD2

Some additional checks

- check 'top hat' electrical isolators are correctly installed at each of the 4x mounting points of the AIDA snout assembly 

- check snout is electrically isolated with respect to AIDA support assembly/stand, bPlas PCBs/cabling/ground/drain wires and BB7

- check LK1 installed aida02 or aida04 *and* aida06 or aida08 ( I assume these are the n+n Ohmic FEE64s? )

- check LK3 installed bottom, middle p+n junction FEE64s ( i.e. LK3 installed on 2 of 16 FEE64 adaptor PCBs )

- check FEE64 power cables are paired correctly, e.g. aida02 & aida04, aida01 & aida03 etc. Remember that the numbering of the power cables at the FEE64 PSUs may no longer correspond to which FEE64 is actually connected to that cable

- check test  and test - cable daisy chains are removed

- check ground cabling attachment to Lemo 00.250 connectors is tight - they have tendency to loosen with handling

Quote:

Mounted on frame:

DSSD 1 (Upstream) :     3208-2/3208-5/3208-8
DSSD 2 (Downstream): 3208-3/3208-21/3208-22

Current mapping has been redone for better management.

AIDA - FEE Mapping
DSSSD 1 DSSSD2
FEE MAC FEE MAC
aida01 41:ba:8a aida06 41:05:15
aida02 41:f6:b7 aida07 41:f6:5a
aida03 41:d8:21 aida08 41:d7:cd
aida04 41:a0:71 aida10 41:d0:0E
aida05 41:cf:ac aida13 41:d8:2b
aida09 41:ee:10 aida14 42:0d:15
aida15 41:b4:0c aida11 41:EE:0f
aida12 41:ba:89 aida16 41:f6:ed

 

Going to try optimising noise now.

 

DHCP updated

new ASIC settings: 2024Mar27-11.25.32 - 16 FEEs (2,4,6,8 n+n, rest p+n)

New layouts: /home/npg/LayOut/GSI_Triple_S100

New layout.txt

 

Firmware of aida11 updated from 0xea40704 to 0x3350706

 

Temps GOOD fig 5

Rates fig 6, 7

Check adapter alignment aida14 and aida16

 

bPlas left/right cables are not insulated and shorting to the snout

Logs on nnpi1 archived and deleted, start again
All 16 FEEs are showing USB logging connectivity and can be monitored with Pi_Monitor

From waveforms aida08 and aida16 are quite unhappy. The rest don't seem too bad. DSSSD 1 is much quieter than it was before!

White Rabbit Analysis: aida02 has lots of WR error counter, HDMI reseat needed
aida09-12 have no WR timestamp, the cable to the MACB is bad or the MACB is bad.
Not needed to fix right now (for noise testing)

Turn off bPlas
We see the noise drop a lot

 

  552   Wed Mar 27 14:22:35 2024 JB, NHInstalling FEE64s of DSSSD2

Mounted on frame:

DSSD 1 (Upstream) :     3208-2/3208-5/3208-8
DSSD 2 (Downstream): 3208-3/3208-21/3208-22

Current mapping has been redone for better management.

AIDA - FEE Mapping
DSSSD 1 DSSSD2
FEE MAC FEE MAC
aida01 41:ba:8a aida06 41:05:15
aida02 41:f6:b7 aida07 41:f6:5a
aida03 41:d8:21 aida08 41:d7:cd
aida04 41:a0:71 aida10 41:d0:0E
aida05 41:cf:ac aida13 41:d8:2b
aida09 41:ee:10 aida14 42:0d:15
aida15 41:b4:0c aida11 41:EE:0f
aida12 41:ba:89 aida16 41:f6:ed

 

Going to try optimising noise now.

 

DHCP updated

new ASIC settings: 2024Mar27-11.25.32 - 16 FEEs (2,4,6,8 n+n, rest p+n)

New layouts: /home/npg/LayOut/GSI_Triple_S100

New layout.txt

 

Firmware of aida11 updated from 0xea40704 to 0x3350706

 

Temps GOOD fig 5

Rates fig 6, 7

Check adapter alignment aida14 and aida16

 

bPlas left/right cables are not insulated and shorting to the snout

Logs on nnpi1 archived and deleted, start again
All 16 FEEs are showing USB logging connectivity and can be monitored with Pi_Monitor

From waveforms aida08 and aida16 are quite unhappy. The rest don't seem too bad. DSSSD 1 is much quieter than it was before!

White Rabbit Analysis: aida02 has lots of WR error counter, HDMI reseat needed
aida09-12 have no WR timestamp, the cable to the MACB is bad or the MACB is bad.
Not needed to fix right now (for noise testing)

Turn off bPlas
We see the noise drop a lot

Attachment 1: Screenshot_2024-03-15_at_14-41-48_Spectrum_Browser_aidas-gsi.png
Screenshot_2024-03-15_at_14-41-48_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2024-03-15_at_14-44-59_Statistics_aidas-gsi.png
Screenshot_2024-03-15_at_14-44-59_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2024-03-15_at_14-55-07_System_wide_Checks_aidas-gsi.png
Screenshot_2024-03-15_at_14-55-07_System_wide_Checks_aidas-gsi.png
Attachment 4: Screenshot_2024-03-15_at_14-57-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2024-03-15_at_14-57-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 5: Screenshot_from_2024-03-27_14-51-12.png
Screenshot_from_2024-03-27_14-51-12.png
Attachment 6: Screenshot_from_2024-03-27_14-53-05.png
Screenshot_from_2024-03-27_14-53-05.png
Attachment 7: Screenshot_from_2024-03-27_14-53-56.png
Screenshot_from_2024-03-27_14-53-56.png
Attachment 8: Screenshot_from_2024-03-27_15-31-10.png
Screenshot_from_2024-03-27_15-31-10.png
Attachment 9: Screenshot_from_2024-03-27_15-35-38.png
Screenshot_from_2024-03-27_15-35-38.png
Attachment 10: Screenshot_from_2024-03-27_15-36-13.png
Screenshot_from_2024-03-27_15-36-13.png
Attachment 11: Screenshot_from_2024-03-27_16-14-44.png
Screenshot_from_2024-03-27_16-14-44.png
Attachment 12: Screenshot_from_2024-03-27_16-15-00.png
Screenshot_from_2024-03-27_16-15-00.png
Attachment 13: Screenshot_from_2024-03-27_16-15-27.png
Screenshot_from_2024-03-27_16-15-27.png
Attachment 14: Screenshot_from_2024-03-27_16-15-56.png
Screenshot_from_2024-03-27_16-15-56.png
  551   Tue Mar 26 18:56:03 2024 TDUSB-controlled ac mains relay interlock box - wiring
Sensor ( 4 pins )

Red  +24V
Blue   0V
Yellow } contact 
Green  } closure in ( short these two wires together for logic 1 )

N.B. yellow and green wires of unused inputs must be connected together


Output ( 3 pins ) to USB-controlled ac mains relay

Red    NC ( normally closed - with respect to COM with no power applied, single pole double throw relay out )
Blue   NO ( normally open - with respect to COM with no power applied, single pole double throw relay out )
Green COM
  550   Tue Mar 26 10:32:05 2024 NH, JBTue 26 March
Taken 4 FEE64s from CRYRING (the 4 easiest to access)
  41:d8:2b
  41:f6:5a
  41:ee:71
  41:d0:0e

In addition a FEE without rails marked only 20 was found in my old office for 5
- This probably came in a shipment to/for repairs

DSSSD#1 unbiased

DESPEC Platform moved out of beam and gamma platform moved from AIDA
JB works on grounding the original 8 DSSSD#1 FEEs

Connected grounding from ribbon cable drain to adapter PCB
Ground loop connected

AIDA rates unchanged bad still fig 1
Waves fig 2 (p+n 6000-9000)
Waves fig 3 (n+n 8000-10000)
Statistics fig 4
Attachment 1: Screenshot_from_2024-03-26_14-39-35.png
Screenshot_from_2024-03-26_14-39-35.png
Attachment 2: Screenshot_from_2024-03-26_14-40-05.png
Screenshot_from_2024-03-26_14-40-05.png
Attachment 3: Screenshot_from_2024-03-26_14-40-30.png
Screenshot_from_2024-03-26_14-40-30.png
Attachment 4: Screenshot_from_2024-03-26_14-41-56.png
Screenshot_from_2024-03-26_14-41-56.png
Attachment 5: 20240326_144539.jpg
20240326_144539.jpg
Attachment 6: 20240326_144638.jpg
20240326_144638.jpg
Attachment 7: 20240326_144528.jpg
20240326_144528.jpg
Attachment 8: 20240326_144659.jpg
20240326_144659.jpg
Attachment 9: 20240326_144627.jpg
20240326_144627.jpg
  549   Fri Mar 22 08:31:39 2024 TDFriday 22 March
09.30 Systems check

      CAEN N1419ET only channel #0 ON, all other channels off
      DSSSD bias -120V leakage current -5.500uA - attachment 1
      Leakage current of 5.5uA corresponds to c. 3nA/cm2/100um indicating high quality device ( assuming ambient temperature c. 21 deg C )

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 2
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida04 WR decoder status 0x10 - attachment 3

      WR timestamps OK - attachment 4

09.42 NH reports "195au implanting in aida, Ca 100 per spill"

      ASIC settings
       LEC/MEC slow comparator 0x64, LEC/MEC fast comparator 0xff, HEC comarator 0x2
       aida02 and aida04 negative input polarity ( n+n Ohmic strips ), all other FEE64s positive input polarity

09.46 all histograms and stats zero'd

      ADC, DISC, PAUSE, RESUME & Correlation Scaler data items stats - attachments 5-9

      per FEE64 1.8.W spectra - 20us FSR - attachments 10-11
       aida08 noise significantly lower than all other FEE64s

      per FEE64 1.8.H spectra - attachments 12-13
       data suggests 195Au ions are focussed on central Si wafer, ion energies to c. 5GeV, no evidence lower A/Z ( fission ) ions with lower energy loss

      per FEE64 1.8.L spectra - attachments 14-15

      per FEE64 Rate  Stat spectra - attachments 16-19

      Merger, TapeServer - attachments 20-21
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

13.30 NH reports "beam over"

      DSSSD bias -120V leakage current -6.500uA - attachment 22

      FEE64 temperatures OK *except* aida02 ASIC temperature - attachment 23
      N.B. aida02 ASIC temperature sensor faulty - reading > 500 deg C - probably poor connection FEE64-ASIC mezzanine

      All system wide checks OK *except*
       aida05 & aida07 FPGA timestamp errors - attachment 24
       aida04 WR decoder status 0x10 - attachment 25

      WR timestamps OK - attachment 26

      ADC data items stats - attachments 27

      per FEE64 Rate spectra - attachments 28-29

      per FEE64 1.8.L spectra - attachments 30-31

      per FEE64 1.8.H spectra - attachments 32-33

      per FEE64 1.8.W spectra - 20us FSR - attachments 34-35

      Merger, TapeServer - attachments 36-37
       Merger idle !?
       Tape Server no storage mode but forwarding data at c. 1Mb/s
       data file R31

20:30 AIDA Powered down

      DSSSD remains biased at -120V, monitor the current over the weekend (Grafana)

      Merger still showing idle... "no data to storage" makes xfer Links disappear?

      Tape server stopped

11.20 Saturday 23 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 38

      https://despec-vm-01.gsi.de/grafana/d/6SAfgl0Mz/aida?orgId=1&refresh=1m&from=now-2d&to=now

      DSSSD#1 leakage current recovered to c. pre-beam values

08.05 Monday 25 March

      Grafana DSSSD bias/leakage current monitor screenshot - attachment 39
Attachment 1: Screenshot_from_2024-03-22_09-32-58.png
Screenshot_from_2024-03-22_09-32-58.png
Attachment 2: Screenshot_from_2024-03-22_09-34-54.png
Screenshot_from_2024-03-22_09-34-54.png
Attachment 3: Screenshot_from_2024-03-22_09-39-03.png
Screenshot_from_2024-03-22_09-39-03.png
Attachment 4: Screenshot_from_2024-03-22_09-40-10.png
Screenshot_from_2024-03-22_09-40-10.png
Attachment 5: Screenshot_from_2024-03-22_09-47-26.png
Screenshot_from_2024-03-22_09-47-26.png
Attachment 6: Screenshot_from_2024-03-22_09-49-33.png
Screenshot_from_2024-03-22_09-49-33.png
Attachment 7: Screenshot_from_2024-03-22_09-49-00.png
Screenshot_from_2024-03-22_09-49-00.png
Attachment 8: Screenshot_from_2024-03-22_09-48-25.png
Screenshot_from_2024-03-22_09-48-25.png
Attachment 9: Screenshot_from_2024-03-22_09-47-55.png
Screenshot_from_2024-03-22_09-47-55.png
Attachment 10: Screenshot_from_2024-03-22_10-00-47.png
Screenshot_from_2024-03-22_10-00-47.png
Attachment 11: Screenshot_from_2024-03-22_10-00-01.png
Screenshot_from_2024-03-22_10-00-01.png
Attachment 12: Screenshot_from_2024-03-22_09-58-56.png
Screenshot_from_2024-03-22_09-58-56.png
Attachment 13: Screenshot_from_2024-03-22_09-57-50.png
Screenshot_from_2024-03-22_09-57-50.png
Attachment 14: Screenshot_from_2024-03-22_09-56-45.png
Screenshot_from_2024-03-22_09-56-45.png
Attachment 15: Screenshot_from_2024-03-22_09-55-52.png
Screenshot_from_2024-03-22_09-55-52.png
Attachment 16: Screenshot_from_2024-03-22_09-54-18.png
Screenshot_from_2024-03-22_09-54-18.png
Attachment 17: Screenshot_from_2024-03-22_09-52-45.png
Screenshot_from_2024-03-22_09-52-45.png
Attachment 18: Screenshot_from_2024-03-22_09-52-13.png
Screenshot_from_2024-03-22_09-52-13.png
Attachment 19: Screenshot_from_2024-03-22_09-51-47.png
Screenshot_from_2024-03-22_09-51-47.png
Attachment 20: Screenshot_from_2024-03-22_10-04-22.png
Screenshot_from_2024-03-22_10-04-22.png
Attachment 21: Screenshot_from_2024-03-22_10-03-53.png
Screenshot_from_2024-03-22_10-03-53.png
Attachment 22: Screenshot_from_2024-03-22_14-29-18.png
Screenshot_from_2024-03-22_14-29-18.png
Attachment 23: Screenshot_from_2024-03-22_14-30-00.png
Screenshot_from_2024-03-22_14-30-00.png
Attachment 24: Screenshot_from_2024-03-22_14-31-19.png
Screenshot_from_2024-03-22_14-31-19.png
Attachment 25: Screenshot_from_2024-03-22_14-31-01.png
Screenshot_from_2024-03-22_14-31-01.png
Attachment 26: Screenshot_from_2024-03-22_14-31-57.png
Screenshot_from_2024-03-22_14-31-57.png
Attachment 27: Screenshot_from_2024-03-22_14-32-35.png
Screenshot_from_2024-03-22_14-32-35.png
Attachment 28: Screenshot_from_2024-03-22_14-40-04.png
Screenshot_from_2024-03-22_14-40-04.png
Attachment 29: Screenshot_from_2024-03-22_14-39-30.png
Screenshot_from_2024-03-22_14-39-30.png
Attachment 30: Screenshot_from_2024-03-22_14-38-22.png
Screenshot_from_2024-03-22_14-38-22.png
Attachment 31: Screenshot_from_2024-03-22_14-37-19.png
Screenshot_from_2024-03-22_14-37-19.png
Attachment 32: Screenshot_from_2024-03-22_14-36-28.png
Screenshot_from_2024-03-22_14-36-28.png
Attachment 33: Screenshot_from_2024-03-22_14-35-23.png
Screenshot_from_2024-03-22_14-35-23.png
Attachment 34: Screenshot_from_2024-03-22_14-34-38.png
Screenshot_from_2024-03-22_14-34-38.png
Attachment 35: Screenshot_from_2024-03-22_14-33-55.png
Screenshot_from_2024-03-22_14-33-55.png
Attachment 36: Screenshot_from_2024-03-22_14-41-21.png
Screenshot_from_2024-03-22_14-41-21.png
Attachment 37: Screenshot_from_2024-03-22_14-40-57.png
Screenshot_from_2024-03-22_14-40-57.png
Attachment 38: Capture_2.PNG
Capture_2.PNG
Attachment 39: Capture_3.PNG
Capture_3.PNG
  547   Fri Mar 22 08:29:55 2024 TDAnydesk restarted remotely
Anydesk restarted remotely per https://elog.ph.ed.ac.uk/CARME/489

Anydesk address now restored to 832827869
  546   Fri Mar 22 08:22:43 2024 NH, AM, MP, CCThu Mar 21
> Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)
> 
> Fig 4-6: After AM and MP turn off Mesytec Preamps
> 
> No difference

I think there is a difference in the 1.8.W spectra - the 100kHz ripple is reduced significantly when the Mesytec preamps are switched off.
  545   Fri Mar 22 07:34:54 2024 NH, JBAu Beam
  544   Thu Mar 21 15:41:49 2024 NH, AM, MP, CCThu Mar 21
Fig 1-3: Noise situation at real thresholds (0xa p+n, 0xf n+n)

Fig 4-6: After AM and MP turn off Mesytec Preamps

No difference
Attachment 1: Screenshot_from_2024-03-21_16-17-59.png
Screenshot_from_2024-03-21_16-17-59.png
Attachment 2: Screenshot_from_2024-03-21_16-18-25.png
Screenshot_from_2024-03-21_16-18-25.png
Attachment 3: Screenshot_from_2024-03-21_16-18-42.png
Screenshot_from_2024-03-21_16-18-42.png
Attachment 4: Screenshot_from_2024-03-21_16-39-12.png
Screenshot_from_2024-03-21_16-39-12.png
Attachment 5: Screenshot_from_2024-03-21_16-39-26.png
Screenshot_from_2024-03-21_16-39-26.png
Attachment 6: Screenshot_from_2024-03-21_16-39-39.png
Screenshot_from_2024-03-21_16-39-39.png
  543   Wed Mar 20 17:02:53 2024 NH/dev/sdd
The following messages are in the system log very often:

Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Mar 20 18:00:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

This (to me) suggests /dev/sdd may be failing. It should be backed up and later replaced (it is not used at the moment for /TapeData, older disk)
  542   Wed Mar 20 12:22:27 2024 NHWed Mar 20

Turn on AIDA for Dry Run demonstrations and so on

All system wide checks, temp, bias OK

Noise situation is dreadful (but has not been optimised). Deterioriation since first mounted, suspect cabling issues with bPlast and BB7.

 Note thresholds at 0x32 (!!!) to not brutalise the DAQs during testing

 aida08 seems OK

Server running to MBS totally fine

18:00

Carole grounded some of the Bplast and this reduced the rates in AIDA, although they are a bit fluctuatey. Due to position constraints she couldn't ground it all
Also AIDA ribbon cables are not grounded yet
The indication is these fixes should make a lot of difference to the situation

AIDA is now powered off for the end of day

Attachment 1: Screenshot_from_2024-03-20_13-20-43.png
Screenshot_from_2024-03-20_13-20-43.png
Attachment 2: Screenshot_from_2024-03-20_13-20-54.png
Screenshot_from_2024-03-20_13-20-54.png
Attachment 3: Screenshot_from_2024-03-20_13-21-18.png
Screenshot_from_2024-03-20_13-21-18.png
Attachment 4: Screenshot_from_2024-03-20_13-21-38.png
Screenshot_from_2024-03-20_13-21-38.png
Attachment 5: Screenshot_from_2024-03-20_13-21-57.png
Screenshot_from_2024-03-20_13-21-57.png
Attachment 6: Screenshot_from_2024-03-20_13-22-12.png
Screenshot_from_2024-03-20_13-22-12.png
Attachment 7: Screenshot_from_2024-03-20_13-24-21.png
Screenshot_from_2024-03-20_13-24-21.png
Attachment 8: Screenshot_from_2024-03-20_13-25-02.png
Screenshot_from_2024-03-20_13-25-02.png
Attachment 9: Screenshot_from_2024-03-20_13-25-13.png
Screenshot_from_2024-03-20_13-25-13.png
  541   Tue Mar 19 10:17:30 2024 NHDry Run 2024 - 19th March 24
AIDA has 8 FEEs and 1 DSSSD 

Aida08 (HDMI#12) had no WR again, I moved it to a different MACB and now it gets WR
The MACB (currently with jsut HDMI#10) seems issues, check/replace the upstream HDMI and thent eh MACB (after dry run!)
 
DSSSD#1 biased to -120 V, leakage current 5.6 uA (fig 1)
 (also on Grafana)

Temps OK fig2

System Checks (fig3-5)
 Clocks OK
 aida07 fails calibration, others OK
 WR OK (aida04 0x10, seems OK)
 FPGA OK

Did "Synchronise ASIC clocks" to align ASIC clocks

(Notes for scalers: SC41L HDMI 5, SCI41R HDMI 9)

18:12 FRS is taking beam, AIDA is powered off and unbiased
Attachment 1: Screenshot_from_2024-03-19_13-44-34.png
Screenshot_from_2024-03-19_13-44-34.png
Attachment 2: Screenshot_from_2024-03-19_13-44-22.png
Screenshot_from_2024-03-19_13-44-22.png
Attachment 3: Screenshot_from_2024-03-19_13-44-04.png
Screenshot_from_2024-03-19_13-44-04.png
Attachment 4: Screenshot_from_2024-03-19_13-43-44.png
Screenshot_from_2024-03-19_13-43-44.png
Attachment 5: Screenshot_from_2024-03-19_13-43-24.png
Screenshot_from_2024-03-19_13-43-24.png
  540   Mon Mar 18 18:04:43 2024 NHPreparation for pre-s100 dry run (and test beam???)
In preparation for the dry run the following *temporary* changes to the FEE numbering have been prepared
These should be reverted after the dry run to ensure cable->fee agreement again

AIDA09 => AIDA06
AIDA11 => AIDA07
AIDA12 => AIDA08

This will allow the merger to run with 8 FEEs for 1 DSSD

dhcpd.conf updated

The 2023Oct19-13.46.30 should work with this numbering (check tomorrow)

As should layout GSI_triple_test_renumber 

AFTER dry run:

Revert DHCP and prepare for full 16 FEEs 

Make new ASIC settings key for 16 FEEs and prepare the aidaXX folders

Prepare a new Layout.mlf set
  539   Fri Mar 15 16:29:57 2024 NHLeakage currents
The behaviour of the DSSSD leakage current at low voltages and during biases is unusual and varies depending on how the adapter boards are connected
To summarise the behaviour I have observed

Minimum bias configuration:
 4 adapter boards, one n+n (LK1), three p+n (-ve bias), ground from n+n to one p+n
 Voltage (and leakage current) unstable at low voltages, seems to settle at around -60 V
 Drops can include 0 leakage current

Full adapter configuration:
 8 adapter boards, ground ring complete
 Same as minimum, but the drops seem to be much smaller (and not to 0 leakage current)
 -60V again seems to be the turnover to a stable leakage current 

In both cases the leakage current during ramping appears basically the same as when settled

Full into FEEs
 8 adapter boards, fully connected to 8 FEEs
 The leakage current is *much* higher during ramp,up to 17 uA near the end. No fluctuations
 Once ramping has finished the current quickly drops back down and settles at the nominal leakage current
 This has been observed in October/December too, it is not new (https://elog.ph.ed.ac.uk/AIDA/910)
 During power up of the FEEs the current sometimes drops briefly (when the ASICs get programmed, I believe)


I think it is related to the ground (more or less current flowing through the HV supply instead of alternate paths?)
It should be kept in mind when testing new detectors to not worry about the detector at low voltages
  537   Thu Mar 14 13:00:23 2024 JB, NH, MA, AM, GAMounting and biasing DSSD 2

new Downstream DSSD2: 3208-2/3208-5/3208-8

Covered with black cloth.

Voltage (V) Current (uA)
10 3.1
20 3.8
30  
40  
50 4.515
60  
70 4.2
80 4.545
90 5.5
100 7.940
110  
120  

Voltage (V)

Current (uA)
10  
20  
30  
40  
50  
60  
70  
100 7.6
110 9.5
120 Breakdown
   
   

Wafer 3 (beam left) shows breakdown issues at 90V... curious

We replace with november beam triple: 3208-3/3208-21/3208-22

We must speak to Micron about why so many 3208 wafers seem to have issues at >90V
It seems unlikely to be other issues

  536   Thu Mar 14 12:13:17 2024 JB, NH, MAMounting and biasing DSSD 1

Upstream bPlast mounted

new Upstream DSSD: 3208-2/3208-5/3208-8

Covered with black cloth.

Voltage (V) Current (uA)
10 3.345
20 3.6
30  
40  
50 4.5ish
60  
70 5.4
80  
90  
100 5.6
110  
120 5.7
   

V-I behaviour found to be nominal.

  535   Wed Mar 13 13:02:19 2024 JB, NH, HA, MADSSD 1 biasing tests

Dismounted Snout and biased DSSSD1 channels

leftmost waifer working

middle waifer reaches 90V then current ramps up

rightmost waifer reaches 80V then current ramps up

DSSSD2 had fingerprint near connections, as shown in the attached image some had been squashed.

Attachment 1: 20240313_142551.jpg
20240313_142551.jpg
  534   Tue Mar 12 16:16:49 2024 NH, TD, JB, HASummary of DSSSD Biasing 12.03
DSSD#1 undergoes a breakdown at 90V, two of the three wafers show this
- The adapter PCBs themselves have no breakdown at 100V, indicating the   issue is internal to the snout

If we are lucky it may be a loose or misaligned kapton connector inside the snout. If not we will have to remove DSSD#2 and inspect DSSD#1 for damage/lint/etc. If we see nothing it should be replaced

DSSD#2 biases perfectly fine, but the leakage current is unstable with biasing -ve to p+n and gnd (return) to n+n. Leakage current is stable biasing +ve to n+n and gnd to p+n. (https://elog.ph.ed.ac.uk/DESPEC/240311_093933/Downstream_positive_bias_vs._Current_(uA).png very nice curve)

We see the same fluctuations just daisy chaining 3 p+n PCBs togerher with the bias (no DSSSD or gnd links). The fluctuations were reduced by connecting all 8 adapter boards of the DSSD together.
When biasing just *one* PCB in this way, the current is stable
The fluctuations happened changing the HV channel, adapter board PCB and LEMO cables: it doesn't seem to be a defect with a specific thing.

The test today confirmed the HV cables (the only ones used) were correctly isolated from everything (OL = "infinite" resistance to ground)

We saw unstable current indications using the Mesytec MHV-4 to apply voltage to the PCB as well.

The behaviour is odd but doesn't seem to be related to the DSSD, which I suspect is OK. One idea is to try connecting the 3 adapter boards to FEEs and repeating the test, as this introduces another (substantial) path to gnd. Maybe this eliminates the current instability? It's about the only difference left from December.

-

For the next steps I believe the snout must be dismounted to inspect DSSD#1. It would be best to coordinate this with replacing the broken(?) bPlas. If we are lucky we may not have to remove the DSSDs but it is likely we have to remove both DSSDs to swap out #1

After replacing the DSSD(s) we should cover the snout with a clean black bin bag and bias it on the MH table to confirm both detectors work. This saves the effort of carrying it to S4 just to find another problem.
If both detectors make it to 120V we can mount it again more confidently

Once we have two biased detectors we can rearrange some FEEs to get 8 for DSSD#1 to do the noise tests. I suggest we wire them up for the numbering plan in https://elog.ph.ed.ac.uk/DESPEC/532 but in the DHCP renumber 9,10,15,16 to 5,6,7,8 temporarily to allow data to be sent to MBS for the dry run (merger limitation)

When the remaining FEEs are recovered from UK+CRYRING we can instrument DSSD#2 and renumber the FEEs to match the cables
  533   Mon Mar 11 15:19:44 2024 JB, NH, Priyanka, Michael Armstrong, Helena AlbersTo Do: AIDA PCB tests

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

 - When the p+n ribbon cables were disconnected for both the upstream and downstream AIDA, the pins and ribbon cables did not appear to have any blemishes.

- Both sets of adaptor PCBs were biased to 100 V and only had an apparent fluctuation in current at c. 0.01 uA. At low voltages (~10V) the current cycled between 0 and 1 uA.

2) Check that all ribbon cables are properly seated in the adaptor PCBs

- When inspected all ribbon cables seemed to be properly seated. A decent press was applied to confirm the seating. The PCBs themselves were inspected and no noticeable damage was observed.

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

- Done for the p+n side, can decide if this is needed for the n+n side.

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

-  Will have to decide further.

5) From email discussion: Check the SHV connector is grounded if one unplugs the cable from the back of the HV module. If yes: We are touching a ground somewhere. If no: it's via the HV (fine).

- When HV#0 was disconnected from the HV module, we used a multimeter on the SHV connecter and frame and read zero resistance 0L on the meter. While when completing the circuit for HV#1 (still) plugged into the HV module we read a resistance of > 1.7 Ohm.

- The HV is touching ground somewhere. This should not be a problem however as the snout is isolated, and this was observed by connecting the SHV to the snout and reading 0L.

6) Properly cover snout with black cloth and bias upstream.

- Snout was covered with black cloth and a black bag. The downstream detector p+n side was biased at 50V and charging/discharging was observed. This is probably due to a short connection somewhere.

Summary: from the p+n side test the voltage-current break down appears to come from inside the snout.

  532   Sun Mar 10 17:08:12 2024 NHAIDA FEE Layout + Cabling Plan for S100
Proposed FEE numbering and wiring plan for upcoming experiment S100 (2x Wide DSSSDs)

Image designed in draw.io, source attached 

FEE numbering is as S450, minimises cable movement from S505/Narrow AIDA
But means merger is not working with 1 DSSD (until all FEEs installed)

Wiring of adapter boards as from noise tests and what should work for DSSD bias
LK3 on middle bottom adapter to ground DSSD
LK1 on one n+n adapter to ground n+n side bias 
p+n has -ve voltage (w.r.t. ground) bias applied via lower adapter boards
ground loop grounds all adapter boards, except 2 p+n adapter boards which are grounded by the bias lemo shield instead

MACB layout also included, with expected NIM logic signals for the aida scalers:

1: Pulser/Sync clock (send to all subsystems, "trigger 3")0

3/4: Time Machine 
5/6: SC41L/R 

All other FEEs have their scaler available
(Scaler should be in left LEMO on MACB, right is output (AIDA->NIM/unused), bottom 4 are triggers from AIDA (unused)

Test circuit will not be used in experiment due to noise, but can be temporarily set up for pulser walkthrough

Revision 2 correct as of 27 March 2023
Attachment 1: AIDA_S100_WiringPlan.drawio.png
AIDA_S100_WiringPlan.drawio.png
Attachment 2: AIDA_S100_WiringPlan.drawio
<mxfile host="app.diagrams.net" modified="2024-03-09T18:35:12.515Z" agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:123.0) Gecko/20100101 Firefox/123.0" etag="cLcX_EiphkJQIDoz3Eud" version="24.0.2" type="device">
  <diagram name="Page-1" id="rdp9_B-ufofv8BSNokWG">
    <mxGraphModel dx="1562" dy="846" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1920" pageHeight="1200" math="0" shadow="0">
      <root>
        <mxCell id="0" />
        <mxCell id="1" parent="0" />
        <mxCell id="Lok4d1QbW70blzeSAxq8-34" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-2" target="Lok4d1QbW70blzeSAxq8-3">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-2" value="9" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="280" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-35" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-3" target="Lok4d1QbW70blzeSAxq8-4">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-3" value="1" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="440" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-4" value="10" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="600" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-42" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-5" target="Lok4d1QbW70blzeSAxq8-6">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-5" value="13" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="280" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-43" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-6" target="Lok4d1QbW70blzeSAxq8-7">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-6" value="5" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="440" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-7" value="14" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="600" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-8" value="15" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="280" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-57" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-9" target="Lok4d1QbW70blzeSAxq8-8">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-9" value="7" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="440" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-40" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-15">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-54" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.75;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-51">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-56" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-9">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-10" value="16" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="600" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-44" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-11" target="Lok4d1QbW70blzeSAxq8-12">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-11" value="11" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="280" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-47" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-12" target="Lok4d1QbW70blzeSAxq8-13">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-12" value="3" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="440" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-32" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-13" target="Lok4d1QbW70blzeSAxq8-14">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-55" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.75;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-13" target="Lok4d1QbW70blzeSAxq8-49">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-13" value="12" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="600" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-61" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-14" target="Lok4d1QbW70blzeSAxq8-2">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-14" value="2" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="160" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-41" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-15" target="Lok4d1QbW70blzeSAxq8-5">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-15" value="6" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="80" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-37" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-16" target="Lok4d1QbW70blzeSAxq8-10">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-16" value="8" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="880" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-17" value="4" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="1">
          <mxGeometry x="800" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-18" value="LK3" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="514" y="416" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-21" value="LK3" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="514" y="496" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-36" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" edge="1" parent="1" source="Lok4d1QbW70blzeSAxq8-22" target="Lok4d1QbW70blzeSAxq8-13">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-22" value="&lt;div&gt;LK1&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="790" y="370" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-23" value="LK1" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="870" y="370" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-49" value="&lt;div&gt;HV CH0&lt;br&gt;NEGATIVE&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="990" y="450" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-51" value="&lt;div&gt;HV CH1&lt;br&gt;NEGATIVE&lt;br&gt;&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="990" y="530" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-59" value="" style="shape=sumEllipse;perimeter=ellipsePerimeter;whiteSpace=wrap;html=1;backgroundOutline=1;" vertex="1" parent="1">
          <mxGeometry x="484" y="305" width="30" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-60" value="Beam Into Page" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="530" y="305" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-62" value="LK1 = Bias -&amp;gt; GND&lt;br&gt;LK3 = DSSD GND -&amp;gt; GND" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" vertex="1" parent="1">
          <mxGeometry x="770" y="125" width="170" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-89" value="" style="group" vertex="1" connectable="0" parent="1">
          <mxGeometry x="230" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-65" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-66" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-67" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-70" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-71" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-72" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-77" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-78" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-79" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-80" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-81" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-84" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-85" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-86" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-87" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-88" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-89">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-90" value="" style="group" vertex="1" connectable="0" parent="1">
          <mxGeometry x="350" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-91" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-92" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-93" value="1" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-94" value="2" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-95" value="3" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-96" value="4" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-97" value="1" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-98" value="2" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-99" value="3" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-100" value="4" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-101" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-102" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-103" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-104" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-105" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-106" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-90">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-107" value="5" style="group" vertex="1" connectable="0" parent="1">
          <mxGeometry x="470" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-108" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-109" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-110" value="5" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-111" value="6" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-112" value="7" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-113" value="8" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-114" value="5" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-115" value="6" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-116" value="7" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-117" value="8" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-118" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-119" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-120" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-121" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-122" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-123" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-107">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-124" value="" style="group" vertex="1" connectable="0" parent="1">
          <mxGeometry x="590" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-125" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-126" value="" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-127" value="9" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-128" value="10" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-129" value="11" style="rounded=0;whiteSpace=wrap;html=1;" vertex="1" parent="Lok4d1QbW70blzeSAxq8-124">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
... 196 more lines ...
Attachment 3: AIDA_S100_WiringPlan(1).drawio
<mxfile host="app.diagrams.net" modified="2024-03-27T12:12:06.799Z" agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:123.0) Gecko/20100101 Firefox/123.0" etag="0KC3XQLCIUXvtcbPW53b" version="23.1.1" type="device">
  <diagram name="Page-1" id="rdp9_B-ufofv8BSNokWG">
    <mxGraphModel dx="794" dy="1153" grid="1" gridSize="10" guides="1" tooltips="1" connect="1" arrows="1" fold="1" page="1" pageScale="1" pageWidth="1920" pageHeight="1200" math="0" shadow="0">
      <root>
        <mxCell id="0" />
        <mxCell id="1" parent="0" />
        <mxCell id="Lok4d1QbW70blzeSAxq8-34" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-2" target="Lok4d1QbW70blzeSAxq8-3" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-2" value="9" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="280" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-35" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-3" target="Lok4d1QbW70blzeSAxq8-4" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-3" value="1" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="440" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-4" value="5" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="600" y="200" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-42" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-5" target="Lok4d1QbW70blzeSAxq8-6" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-5" value="10" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="280" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-43" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-6" target="Lok4d1QbW70blzeSAxq8-7" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-6" value="14" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="440" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-7" value="13" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="600" y="120" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-8" value="11" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="280" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-57" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;" parent="1" source="Lok4d1QbW70blzeSAxq8-9" target="Lok4d1QbW70blzeSAxq8-8" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-9" value="7" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="440" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-40" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-15" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-54" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.75;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-51" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-56" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0;exitY=0.5;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;" parent="1" source="Lok4d1QbW70blzeSAxq8-10" target="Lok4d1QbW70blzeSAxq8-9" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-10" value="16" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="600" y="480" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-44" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-11" target="Lok4d1QbW70blzeSAxq8-12" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-11" value="15" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="280" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-47" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=1;exitY=0.5;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-12" target="Lok4d1QbW70blzeSAxq8-13" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-12" value="3" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="440" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-32" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=0.5;entryY=1;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-13" target="Lok4d1QbW70blzeSAxq8-14" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-55" value="BIAS" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.75;exitY=1;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#ffe6cc;strokeColor=#d79b00;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-13" target="Lok4d1QbW70blzeSAxq8-49" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-13" value="12" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="600" y="400" width="120" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-61" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-14" target="Lok4d1QbW70blzeSAxq8-2" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-14" value="2" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="160" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-41" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=0;exitDx=0;exitDy=0;entryX=0;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-15" target="Lok4d1QbW70blzeSAxq8-5" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-15" value="6" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="80" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-37" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-16" target="Lok4d1QbW70blzeSAxq8-10" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-16" value="8" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="880" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-17" value="4" style="rounded=0;whiteSpace=wrap;html=1;" parent="1" vertex="1">
          <mxGeometry x="800" y="240" width="40" height="160" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-18" value="LK3" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="514" y="416" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-21" value="LK3" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="514" y="496" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-36" value="GND" style="edgeStyle=orthogonalEdgeStyle;rounded=0;orthogonalLoop=1;jettySize=auto;html=1;exitX=0.5;exitY=1;exitDx=0;exitDy=0;entryX=1;entryY=0.5;entryDx=0;entryDy=0;fillColor=#d5e8d4;strokeColor=#82b366;endArrow=none;endFill=0;strokeWidth=2;" parent="1" source="Lok4d1QbW70blzeSAxq8-22" target="Lok4d1QbW70blzeSAxq8-13" edge="1">
          <mxGeometry relative="1" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-22" value="&lt;div&gt;LK1&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="790" y="370" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-23" value="LK1" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="870" y="370" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-49" value="&lt;div&gt;HV CH0&lt;br&gt;NEGATIVE&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="990" y="450" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-51" value="&lt;div&gt;HV CH1&lt;br&gt;NEGATIVE&lt;br&gt;&lt;/div&gt;" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="990" y="530" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-59" value="" style="shape=sumEllipse;perimeter=ellipsePerimeter;whiteSpace=wrap;html=1;backgroundOutline=1;" parent="1" vertex="1">
          <mxGeometry x="484" y="305" width="30" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-60" value="Beam Into Page" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="530" y="305" width="60" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-62" value="LK1 = Bias -&amp;gt; GND&lt;br&gt;LK3 = DSSD PCB GND -&amp;gt; GND" style="text;html=1;align=center;verticalAlign=middle;whiteSpace=wrap;rounded=0;" parent="1" vertex="1">
          <mxGeometry x="760" y="125" width="180" height="30" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-89" value="" style="group" parent="1" vertex="1" connectable="0">
          <mxGeometry x="230" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-65" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-66" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-67" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-70" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-71" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-72" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-77" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-78" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-79" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-80" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-81" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-84" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-85" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-86" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-87" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-88" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-89" vertex="1">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-90" value="" style="group" parent="1" vertex="1" connectable="0">
          <mxGeometry x="350" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-91" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-92" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-93" value="1" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-94" value="2" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-95" value="3" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-96" value="4" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-97" value="1" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-98" value="2" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-99" value="3" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-100" value="4" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-101" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-102" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-103" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-104" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-105" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-106" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-90" vertex="1">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-107" value="5" style="group" parent="1" vertex="1" connectable="0">
          <mxGeometry x="470" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-108" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-109" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-110" value="5" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-111" value="6" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-112" value="7" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-113" value="8" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="25" y="230" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-114" value="5" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-115" value="6" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-116" value="7" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-117" value="8" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-118" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="10" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-119" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="310" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-120" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="340" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-121" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="370" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-122" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="400" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-123" value="" style="ellipse;whiteSpace=wrap;html=1;aspect=fixed;" parent="Lok4d1QbW70blzeSAxq8-107" vertex="1">
          <mxGeometry x="40" y="440" width="20" height="20" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-124" value="" style="group" parent="1" vertex="1" connectable="0">
          <mxGeometry x="590" y="590" width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-125" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry width="70" height="480" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-126" value="" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry x="25" y="30" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-127" value="9" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry x="25" y="80" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-128" value="10" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry x="25" y="130" width="20" height="40" as="geometry" />
        </mxCell>
        <mxCell id="Lok4d1QbW70blzeSAxq8-129" value="11" style="rounded=0;whiteSpace=wrap;html=1;" parent="Lok4d1QbW70blzeSAxq8-124" vertex="1">
          <mxGeometry x="25" y="180" width="20" height="40" as="geometry" />
        </mxCell>
... 207 more lines ...
Attachment 4: AIDA_S100_WiringPlan.drawio.png
AIDA_S100_WiringPlan.drawio.png
  531   Fri Mar 8 16:05:57 2024 JB, CC, TD, NHFriday 8 March

Bias tests of AIDA on individual wafers and in parallel. Spreadsheet can be found in attachment 1, and graphic results can be seen in Attachment 2. Summary: Upstream detector cannot be biased in parallel nor individually. Downstream detector can be biased with positively and each wafer can individually be biased negatively. In parallel, it was not possible to bias the downstream detector negatively as indicated by the tests below.

Test downstream DSSSD with positive polarity bias

Configuration as follows:

CAEN N1419ET ch #3 connected to LHS FEE64 adaptor PCB ( looking upstream ) - LK1 *not* fitted LK1 fitted to 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cables from/to GND terminals of 3x ( top )

FEE64 adaptor PCBs *and* LHS FEE64 adaptor PCB 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything, LK3 fitted Total 5x adaptor PCBs installed

No other LKs fitted

Bias Voltage (V) Current (uA)
+10 2.150
+20 3.300
+30 4.035
+40 4.550
+50 4.955
+60 5.255
+70 5.490
+80 5.650
+90 5.730
+100 5.780
+110 5.825
+120 5.860

Nominal V-I curve, stable leakage current. Attachment 3.

Following this success we attempted to repeat test using negative polarity bias

Configuration as follows:

CAEN N1419ET ch #1 connected to ( top, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( top )

FEE64 adaptor PCBs Lemo 00.250 jumper cable from/to GND terminals of LHS and ( top, left )

FEE64 adaptor PCBs 1x ( bottom, middle )

FEE64 adaptor PCB connected to ribbon cables but not otherwise connected to anything,

LK3 fitted Total 5x adaptor PCBs installed No other LKs fitted With detector bias

-20V we continue to observe the leakage current cycling between 0 and ~2uA with a frequency ~1Hz ( as before )

Copy configuration used for upstream DSSSD test ( which was successful albeit there was detector breakdown at bias voltages > c. 90V )

CAEN N1419ET ch #1 connected to ( bottom, left )

FEE64 adaptor PCB ( looking upstream ) LHS

FEE64 adaptor PCB ( looking upstream ) - LK1 fitted Lemo 00.250 jumper cables from/to BIAS terminals of 3x ( bottom )

FEE64 adaptor PCBs Ground cable jumpered from/to GND terminals LHS, ( left, bottom ), RHS and all 3x top FEE64 adaptor PCBs ( bottom, middle )

FEE64 adaptor PCB LK3 fitted Total 8x adaptor PCBs installed

No other LKs fitted With detector bias -20V we observe leakage current of ~2-3uA.

Current unstable - variations 10-100nA over periods of several seconds Although the leakage current is unstable this is an improvement over previous tests with negative bias. The duplication of upstream and downstream configurations suggests that for some unknown reason it is necessary to connect all 8x FEE64 adaptor PCBs whereas our expectation was that only 4x were necessary.

Summary: Upstream DSSSD Si wafers 1 & 2 breakdown for bias > c. 90V Si wafer 3 OK to 120V Positive bias - not tested Negative bias OK - leakage current stable to c. 90V Downstream DSSSD Si wafers 1, 2 & 3 OK to 120V Positive bias OK Negative bias - leakage current unstable

To do:

1) Disconnect ribbon cables from p+n junction FEE64 adaptor PCBs of upstream DSSSD, apply c. 100V bias and check leakage current is zero i.e. eliminate shorts in PCBs

2) Check that all ribbon cables are properly seated in the adaptor PCBs

3) Check all adaptor PCB connector pins are OK: will need to remove ribbon cables

4) If/when you open check seating of all Kapton PCBs in the DSSSD connectors *and* carefully check that ribbon cable and Kapton PCB connectors are aligned and not out by 1 or 2 rows say.

 

Attachment 1: IV_test_AIDA.xlsx
Attachment 2: chart.png
chart.png
Attachment 3: Downstream_positive_bias_vs._Current_(uA).png
Downstream_positive_bias_vs._Current_(uA).png
  530   Thu Mar 7 15:15:29 2024 JB, NH, CCAIDA HV Bias Test IV Curve
AIDA I-V Test
Voltage (V) Current (uA)
10 1.43
20 2.065
30 2.415
40 2.64
50 2.825
60 2.99
70 3.185
80 3.58
90 5.01
100 9.4

- Voltage-Current test of newly installed AIDA snout. Breakdown observed at around ~92 V. Probably caused my light leakage into the snout, will investigate further.

- Tried to cover the snout with a black cloth. This did not change the breakdown behaviour.

  529   Thu Mar 7 11:09:05 2024 HA, JB, CC, TD, MGThursday 7 March
Snout assembly

DSSSDs

Upstream   3208-10/3208-18/3208-20
Downstream 3131-5/3131-10/3131-12

p+n junction side bias continuity checked OK for *all* wafers
 all c. 22 Ohm as expected

n+n Ohmic side bias continuity checked OK for both DSSSDs
 upstream 73 Ohm, downstream 92 Ohm

n+n Ohmic side bias wafer #2 
 positioned at top of snout assembly ( lower stage snout side marked 'T' )

p+n junction side faces downstream


Distances

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.0cm
ref - upstream DSSSD 11.0cm
ref - downstream DSSSD 12.0cm
ref - middle of downstream bPlas 14.0cm

Measurements consistent to +/-1mm between LHS and RHS of snout assembly as viewed from top side 


13.00 At this position the Kapton PCBs bulged in/out and there was some concern that when the upper stage
snout was installed it would the Kapton PCBs into the 'active' area. It was decided to move the 2x bPlas
and 2x DSSSDs c. 5mm downstream.

Jeroen and Phillip cut sections from 4x PEEK 6mm spacers so that they would clip onto 3mm dia support rods.

We raised the assembly fro the inter stage and inserted the modified spacers between the inter stage and 
the other spacers above. The Kapton PCBs now run fairly straight to the DSSSD connectors.

Distances re-measured as

Using top edge of lower stage snout as reference

ref - middle of upstream bPlas 8.6cm
ref - upstream DSSSD 11.5cm
ref - downstream DSSSD 12.4cm
ref - middle of downstream bPlas 14.5cm
  528   Tue Mar 5 09:10:01 2024 MA, JB, TDTuesday 5 March
Old ribbon cable assemblies - attachments 1  2

Length of cable from base of AIDA snout assembly to Samtec FFSD ribbon cable connectors

p+n junction side 10-11cm & 25.5cm
n+n Ohmic side 11cm & 25cm

Distance from Delron base of AIDA snout assembly to Samtec CLP connectors ( Kapton PCBs ) 51.5cm & 52.5cm



Manufacture of new ribbon cable assemblies - attachment 3

8x 2x 29" Samtec FFSD 34-way cables
8x 2x 23" Samtec FFSD 34-way cables


17.52 from Helena

"These are the detector positions within the snout:

Total snout length of 573 mm (380 mm bottom stage + 193 top stage), relative to the 'black flange' of the AIDA frame

Upstream AIDA @ 513 mm

Downstream AIDA @ 523 mm

This leaves 5cm space for the Downstream bPlast AND the BB7 layer together. I believe this should be enough space - can you all confirm for AIDA/bPlast/BB7 if this is 
agreed upon? It would be great to get a fast response so that we are ready for mounting tomorrow. If we need more space we could think of shifting everything upstream by 
10mm.

The upstream bPlast will need to go first such that the upstream AIDA is at 513mm. I don't have the measurements to hand to give the position."
Attachment 1: 20240305_101140.jpg
20240305_101140.jpg
Attachment 2: 20240305_101146.jpg
20240305_101146.jpg
Attachment 3: 20240305_165949.jpg
20240305_165949.jpg
  527   Mon Mar 4 11:35:30 2024 TDMonday 4 March
12.30 FEE64 41:d7:cd ASIC mezzanine u/s, no data, ASIC temperature c. 20 deg C ( ambient ) low
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida08

      Water temperature & pressure as measured outside S4 area - OK 

      Manually power FEE64s

      BNC PB-5 local control/ON
      Amplitude 1.0V
      Attenuation x10
      Polarity +
      tau_d 1ms
      Frequency 22Hz

      FEE64 temps - attachment 1
       aida02 with new ASIC mezzanine continues to ramp to c. 512 deg C - initially reads 0, next refresh c. 70 and third refresh c. 512 deg C
       aida02 Virtex and PSU temps OK, aida02 cooling plate ambient to touch

       aida08 ASIC temp c. 45 deg C as expected, Virtex and PSU temps OK

      All system wide checks OK

      WR timestamp OK - attachment 2

      aida04 ASIC settings - attachment 3

      ADC data item stats OK - attachment 4
   
      aida04 Rate spectrum - attachment 5

      aida04 *.*.L spectra - attachments 6-9
       1.8.L pulser peak width 13 ch FWHM

      aida04 1.8.W spectra - 20us FSR - attachment 10-11
Attachment 1: Screenshot_from_2024-03-04_12-27-04.png
Screenshot_from_2024-03-04_12-27-04.png
Attachment 2: Screenshot_from_2024-03-04_12-37-24.png
Screenshot_from_2024-03-04_12-37-24.png
Attachment 3: Screenshot_from_2024-03-04_12-27-34.png
Screenshot_from_2024-03-04_12-27-34.png
Attachment 4: Screenshot_from_2024-03-03_19-02-33.png
Screenshot_from_2024-03-03_19-02-33.png
Attachment 5: Screenshot_from_2024-03-04_12-29-44.png
Screenshot_from_2024-03-04_12-29-44.png
Attachment 6: Screenshot_from_2024-03-04_12-33-43.png
Screenshot_from_2024-03-04_12-33-43.png
Attachment 7: Screenshot_from_2024-03-04_12-32-50.png
Screenshot_from_2024-03-04_12-32-50.png
Attachment 8: Screenshot_from_2024-03-04_12-32-11.png
Screenshot_from_2024-03-04_12-32-11.png
Attachment 9: Screenshot_from_2024-03-04_12-31-30.png
Screenshot_from_2024-03-04_12-31-30.png
Attachment 10: Screenshot_from_2024-03-04_12-34-49.png
Screenshot_from_2024-03-04_12-34-49.png
Attachment 11: Screenshot_from_2024-03-04_12-34-40.png
Screenshot_from_2024-03-04_12-34-40.png
  526   Sun Mar 3 16:06:11 2024 TDSunday 3 March
17.00 FEE64 41:a0:71 ASIC #2 u/s
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida04

      Unable to boot nnrpi2 - manual/local control of BNC PB-5 and FEE64 PSU - no interlock

      Water temperature & pressure as measured outside S4 area - OK 

      Manually power FEE64s

      BNC PB-5 local control/ON
      Amplitude 1.0V
      Attenuation x10
      Polarity -
      tau_d 1ms
      Frequency 22Hz

      FEE64 temps OK - attachment 1

      All system wide checks OK

      WR timestamp OK - attachment 2

      aida04 ASIC settings - attachment 3

      ADC data item stats OK - attachment 4
   
      aida04 Rate spectrum - attachment 5

      aida04 *.*.L spectra - attachments 6-9
       1.8.L pulser peak width 15 ch FWHM

      aida04 1.8.W spectra - 20us FSR - attachment 10-11

      MIDAS configuration - attachment 12

      DHCP config - attachment 13



18.30 FEE64 41:f6:b7 ASIC #1 u/s
      Replaced ASIC mezzanine with new ASIC mezzanine
      Installed as aida02

      All system wide checks OK

      FEE64 temps - attachment 14
       aida02 ASIC temperature c. 512 deg C !
       PSU/Virtex temps OK
       Mezzanine ambient to touch - assume faulty sensor/poor connection?

      aida02 ASIC settings - attachment 15

      WR timestamp OK - attachment 16

      ADC data item stats OK - attachment 4

      aida04 Rate spectrum - attachment 18

      aida02 1.8.W spectra - 20us FSR - attachment 19-20

      aida02 *.*.L spectra - attachments 21-24
       1.8.L pulser peak width 16 ch FWHM


19.30 Power OFF



     
Attachment 1: Screenshot_from_2024-03-03_16-54-42.png
Screenshot_from_2024-03-03_16-54-42.png
Attachment 2: Screenshot_from_2024-03-03_16-55-14.png
Screenshot_from_2024-03-03_16-55-14.png
Attachment 3: Screenshot_from_2024-03-03_16-55-21.png
Screenshot_from_2024-03-03_16-55-21.png
Attachment 4: Screenshot_from_2024-03-03_17-07-26.png
Screenshot_from_2024-03-03_17-07-26.png
Attachment 5: Screenshot_from_2024-03-03_16-53-52.png
Screenshot_from_2024-03-03_16-53-52.png
Attachment 6: Screenshot_from_2024-03-03_17-02-09.png
Screenshot_from_2024-03-03_17-02-09.png
Attachment 7: Screenshot_from_2024-03-03_17-01-52.png
Screenshot_from_2024-03-03_17-01-52.png
Attachment 8: Screenshot_from_2024-03-03_17-00-18.png
Screenshot_from_2024-03-03_17-00-18.png
Attachment 9: Screenshot_from_2024-03-03_16-59-23.png
Screenshot_from_2024-03-03_16-59-23.png
Attachment 10: Screenshot_from_2024-03-03_16-58-43.png
Screenshot_from_2024-03-03_16-58-43.png
Attachment 11: Screenshot_from_2024-03-03_16-57-09.png
Screenshot_from_2024-03-03_16-57-09.png
Attachment 12: startup.tcl

   Setting BGColor "BlanchedAlmond"
   Setting TextColor "red"

      Setting ProjectName  AIDA
      Setting ProjectImage AIDA.gif
      Setting ProjectURL   "http://npg.dl.ac.uk/NPG/Projects/Fair.html"


  Setting Option1Name STFC
  Setting Option1URL  "stfc.ac.uk"
  Setting Option1Image stfc.jpg


# global MessageLoggerServerHost
#  set  MessageLoggerServerHost nndhcp077.dl.ac.uk

#    package require  MessageLoggerClient 

#  Setting Option3Name "Elog Server"
#  Setting Option3URL  "http://nndhcp077.dl.ac.uk:8080/DataAcq/"

#    Install required services

    if {![namespace exists ::AIDA]} {namespace eval ::AIDA {puts "created namespace ::AIDA"}}
#    set ::AIDA::TestMode 1

    global SpectrumServerHost
    set SpectrumServerHost aidas

   package require XAIDAAccessClient  ;# for Client end - ie AIDA server workstation  sets MIDAS_XAIDAAccessClient_Exists (SOAP)

   package require XAD9252AccessClient ;# for Client end - ie AIDA server workstation  sets MIDAS_XAD9252AccessClient_Exists (SOAP)

namespace eval DataAcquisition {
     variable PROJECT AIDA
     variable ACQSERVERS
     set ACQSERVERS [list aida01 aida03 aida04 aida05 aida08]
     variable ACQSERVER [first $ACQSERVERS]
     variable DataTransferStreams; set DataTransferStreams 2
#     set ::DataAcquisition::EnableMerger 1
#     set ::DataAcquisition::EnableTape 1
    variable RDOGo 0 
}

namespace eval AIDA {
      variable ModuleMenu; set ModuleMenu $::DataAcquisition::ACQSERVERS
      variable TSMaster; set TSMaster aida01
      variable FEE64 [first $ModuleMenu]
      variable ConfigBase; set ConfigBase [file join $env(MIDASBASE) config TclHttpd]
}

    Setting DataBaseRoot "/MIDAS/DB"
    package require DataBaseAccessServer

    package require NetVarsServer 1.1
    package require SigTaskServer

#    package require MemSasServer

#    Install services for Run Control

namespace eval $::DataAcquisition::PROJECT {
    variable DBRootName [file join EXPERIMENTS AIDA]
    variable DBNodeBase 2012
}

  package require AIDARunControl
#  package require TapeServer
#  package require MergerControl
#  package require MERGERunControl


  Setting Option7Name "Project Home Page"
  Setting Option7URL  "http://npg.dl.ac.uk/NPG/Projects/Fair.html"

  Setting Option8Name "Server Software Home Page"
  Setting Option8URL  "http://npg.dl.ac.uk/MIDAS/download/WebServices.html"


Attachment 13: dhcpd.conf
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
# Date of last update  Jan 12 2015 
#
authoritative;

ddns-update-style none; ddns-updates off;

# 2 days
#default-lease-time 172800;
# 4 days
default-lease-time 345600;
# 8 days
max-lease-time 691200;

option domain-search code 119 = string;

option domain-name "dl.ac.uk";
option domain-name-servers 193.62.115.16, 148.79.80.78;
option netbios-name-servers 148.79.160.89;
option netbios-node-type 8;
option nis-domain "nuclear.physics";
option nis-servers 193.62.115.77;


subnet 192.168.11.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.11.255;
   pool {
 range 192.168.11.118 192.168.11.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.11.99";
   next-server 192.168.11.99;

   host nnrpi1 {
     hardware ethernet b8:27:eb:bb:46:7b;
     fixed-address 192.168.11.251;
   }

   host nnrpi2 {
     hardware ethernet b8:27:eb:40:53:e8;
     fixed-address 192.168.11.117;
   }

   host edinburgh {
      hardware ethernet 00:14:4f:9b:6e:7a;
      fixed-address 192.168.11.118;
   }

   host aida01 {
     hardware ethernet d8:80:39:41:ba:8a;
     fixed-address 192.168.11.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01";
   }

   host aida02 {
     hardware ethernet d8:80:39:41:ba:22;
     fixed-address 192.168.11.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02";
   }

   host aida03 {
     hardware ethernet d8:80:39:41:d8:21;
     fixed-address 192.168.11.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03";
   }

   host aida04 {
     hardware ethernet d8:80:39:41:a0:71;
     fixed-address 192.168.11.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04";
   }


   host aida05 {
     hardware ethernet d8:80:39:41:ee:10;
     fixed-address 192.168.11.5;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   }

   host aida06 {
     hardware ethernet d8:80:39:41:cf:ac;
     fixed-address 192.168.11.6;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   }

   host aida07 {
     hardware ethernet d8:80:39:41:f6:5a;
     fixed-address 192.168.11.7;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   }

   host aida08 {
     hardware ethernet d8:80:39:41:ba:89;
     fixed-address 192.168.11.8;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   }

   host aida09 {
     hardware ethernet d8:80:39:42:0d:15;
     fixed-address 192.168.11.9;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09";
   }

   #host aida06 {
   #  hardware ethernet d8:80:39:41:ee:72;
   #  fixed-address 192.168.11.6;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   #}

   #host aida07 {
   #  hardware ethernet d8:80:39:41:b4:0c;
   #  fixed-address 192.168.11.7;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   #}

   #host aida08 {
   #  hardware ethernet d8:80:39:41:ba:2b;
   #  fixed-address 192.168.11.8;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   #}

   #host aida13 {
   #  hardware ethernet d8:80:39:42:d:15;
   #  fixed-address 192.168.11.13;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida13";
   #}

#   host aida14 {
#     hardware ethernet d8:80:39:42:d:b;
#     fixed-address 192.168.11.14;
#    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14";
#   }

   #host aida05 {
   #  hardware ethernet d8:80:39:41:ee:10;
   #  fixed-address 192.168.11.5;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   #}

   #host aida14 {
   #  hardware ethernet d8:80:39:41:f6:ed;
   #  fixed-address 192.168.11.14;
   # option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14";
   #}
}


subnet 192.168.12.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.12.255;
   pool {
 range 192.168.12.100 192.168.12.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.12.99";
   next-server 192.168.12.99;







}


Attachment 14: Screenshot_from_2024-03-03_18-53-03.png
Screenshot_from_2024-03-03_18-53-03.png
Attachment 15: Screenshot_from_2024-03-03_18-53-16.png
Screenshot_from_2024-03-03_18-53-16.png
Attachment 16: Screenshot_from_2024-03-03_18-53-41.png
Screenshot_from_2024-03-03_18-53-41.png
Attachment 17: Screenshot_from_2024-03-03_19-02-33.png
Screenshot_from_2024-03-03_19-02-33.png
Attachment 18: Screenshot_from_2024-03-03_18-53-47.png
Screenshot_from_2024-03-03_18-53-47.png
Attachment 19: Screenshot_from_2024-03-03_19-01-41.png
Screenshot_from_2024-03-03_19-01-41.png
Attachment 20: Screenshot_from_2024-03-03_19-01-31.png
Screenshot_from_2024-03-03_19-01-31.png
Attachment 21: Screenshot_from_2024-03-03_19-00-11.png
Screenshot_from_2024-03-03_19-00-11.png
Attachment 22: Screenshot_from_2024-03-03_18-59-24.png
Screenshot_from_2024-03-03_18-59-24.png
Attachment 23: Screenshot_from_2024-03-03_18-58-39.png
Screenshot_from_2024-03-03_18-58-39.png
Attachment 24: Screenshot_from_2024-03-03_18-57-57.png
Screenshot_from_2024-03-03_18-57-57.png
  525   Mon Feb 26 13:17:51 2024 TDOffline analysis of DEC23/R9_6
First pass analysis of data file /TapeData/DEC23/R9_6

 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  234532726 (   99567.1 Hz)
        Other data format:   27387282 (   11626.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         53 (       0.0 Hz)
                               RESUME:         53 (       0.0 Hz)
                              SYNC100:      40775 (      17.3 Hz)
                              WR48-63:      40775 (      17.3 Hz)
                           FEE64 disc:    2928025 (    1243.0 Hz)
                             MBS info:   24377601 (   10349.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     287073 (     121.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2355.524 s

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   59013868     372005          0          0         20         20       9888       9888     281563      70626          0      55467
  1   31445079    1098714          0          0          2          2       5017       5017     243009     845667          0      38876
  2   22895499   11841669          0          0          1          1       5294       5294     482655   11348424          0      42748
  3   18228498   11705354          0          0          0          0       3913       3913     416619   11280909          0      78526
  4   25500772    1237178          0          0          3          3       4320       4320     396557     831975          0      28918
  5          4          0          0          0          0          0          0          0          0          0          0          0
  6   15933921     394210          0          0          0          0       2560       2560     389090          0          0      11489
  7   61515085     738152          0          0         27         27       9783       9783     718532          0          0      31049

FEE64 configuration https://elog.ph.ed.ac.uk/AIDA/935


Note

- aida06 not connected to DSSSD ( cabling broken  - will be replaced later this week )

- LEC ( 20MeV FSR ) data ADC offset corrected 
   ADC offsets available for 489 of 512 channels and are included in the analysis - 23 strips for which no ADC offsets could be calculated ( usually because there was 
no 
pulser data ) are not included in analysis 
- LEC ( 20MeV FSR ) front-back strip energy difference cut +/- 50 channels ( c. +/- 280keV )
- HEC ( 2GeV FSR ) front-back strip energy difference cut +/-200 channels ( c. +/- 1120MeV )

- LEC ADC data: 13 < channel < 188 ( c. 73-1053keV )
- HEC ADC data: > 13 channels ( c. 73MeV )
- FEE64 hardware thresholds: LEC c. 100keV, HEC c. 200MeV 




Attachments 1 & 2 - per pixel HEC-LEC event time difference spectra - 4.096us/channel

Attachments 3 & 4 - per pixel HEC-LEC event time difference spectra - 65.536us/channel

Attachment 5 - DSSSD x-y hit pattern: HEC-LEC event time difference < 4.3s
 Majority of events associated with 'hot' p+n junction strips - few plausible decay candidate events - as expected.
 z-scale - semi-logarithmic - scattered events are single counts ( blue )
 1 count => rate ~ 1/2355s ~ 0.0004Hz, the great majority of pixels have zero counts

Attachment 6 - LEC ( decay ) and HEC ( implant ) events - 262.144us/channel ( 65536 channels = 17.2s ) 

HEC implants channels 12000-25000 = 216 counts => HEC rate ( in spill ) 63.4Hz
LEC decays channels 12000-25000 = 3865 counts => LEC rate ( in spill ) 1134.1Hz
LEC decays channels 25000-38000 = 2790 counts => LEC rate ( inter spill ) 818.7Hz

Attachment 7 - per pixel HEC-HEC event time difference spectra - 4.096us/channel

Attachment 8 - per pixel LEC-LEC event time difference spectra - 4.096us/channel

Attachment 9 - variables.dat
This is a ( Fortran ) Namelist I/O data file containing of the ADC offsets, FEE64 configuration, LEC and HEC energy difference windows 

The ADC offset channel number is calculated as

channel = channel_ident + ( module * 64 ) where module = 0-7 corresponding to AIDA FEE64s aida01-aida08

and is used as follows

ADC data = INT ( RSHIFT( ABS( 32768 - data( i ) ), 3 ) - offset( i ) + 0.5 )

where data(i) is the ADC data item for channel i, offset(i) is the ADC offset for channel i

An ADC offset of -9999 means there was no pulser data for this channel in data files R5 and R11.

               
Attachment 1: Screenshot_from_2024-03-01_11-41-11.png
Screenshot_from_2024-03-01_11-41-11.png
Attachment 2: Screenshot_from_2024-03-01_11-41-30.png
Screenshot_from_2024-03-01_11-41-30.png
Attachment 3: Screenshot_from_2024-03-01_11-42-06.png
Screenshot_from_2024-03-01_11-42-06.png
Attachment 4: Screenshot_from_2024-03-01_11-42-22.png
Screenshot_from_2024-03-01_11-42-22.png
Attachment 5: Screenshot_from_2024-03-01_11-43-13.png
Screenshot_from_2024-03-01_11-43-13.png
Attachment 6: Screenshot_from_2024-03-01_11-44-33.png
Screenshot_from_2024-03-01_11-44-33.png
Attachment 7: Screenshot_from_2024-03-01_11-38-52.png
Screenshot_from_2024-03-01_11-38-52.png
Attachment 8: Screenshot_from_2024-03-01_11-40-37.png
Screenshot_from_2024-03-01_11-40-37.png
Attachment 9: variables.dat
  $variables
  offset(   0) =    10.71
  offset(   1) =    14.56
  offset(   2) =   -13.88
  offset(   3) =    -9.85
  offset(   4) =   -18.84
  offset(   5) =    -3.41
  offset(   6) =    24.00
  offset(   7) =    23.10
  offset(   8) =    -2.39
  offset(   9) =   -15.90
  offset(  10) =    25.89
  offset(  11) =   -55.27
  offset(  12) =     6.39
  offset(  13) =   -31.83
  offset(  14) =     7.94
  offset(  15) =     7.69
  offset(  16) =     3.46
  offset(  17) =    -3.12
  offset(  18) =    16.24
  offset(  19) =    -6.69
  offset(  20) =    18.42
  offset(  21) = -9999.99
  offset(  22) =   -22.39
  offset(  23) =     1.94
  offset(  24) =   -11.29
  offset(  25) =    -5.55
  offset(  26) =     7.88
  offset(  27) =    -4.32
  offset(  28) =    -7.79
  offset(  29) =   -13.60
  offset(  30) =     2.61
  offset(  31) =    14.95
  offset(  32) =     4.43
  offset(  33) = -9999.99
  offset(  34) =    -6.61
  offset(  35) =   -22.58
  offset(  36) = -9999.99
  offset(  37) =    11.90
  offset(  38) = -9999.99
  offset(  39) =    -1.80
  offset(  40) =   -26.27
  offset(  41) = -9999.99
  offset(  42) =    -1.99
  offset(  43) =    -3.66
  offset(  44) = -9999.99
  offset(  45) =     7.10
  offset(  46) =    -0.31
  offset(  47) =    26.80
  offset(  48) =   -16.60
  offset(  49) = -9999.99
  offset(  50) =    -9.08
  offset(  51) =   -10.53
  offset(  52) =   -28.24
  offset(  53) =    13.09
  offset(  54) =   -12.42
  offset(  55) =   -22.52
  offset(  56) =     3.27
  offset(  57) =   -26.47
  offset(  58) =     7.17
  offset(  59) =     5.31
  offset(  60) =   -29.79
  offset(  61) =    19.75
  offset(  62) =     2.12
  offset(  63) =   -22.37
  offset(  64) =     3.02
  offset(  65) =     1.74
  offset(  66) =    23.39
  offset(  67) =    43.87
  offset(  68) =   -15.18
  offset(  69) =    11.78
  offset(  70) =   -14.59
  offset(  71) =     1.89
  offset(  72) =   -15.75
  offset(  73) =    25.22
  offset(  74) =     7.83
  offset(  75) = -9999.99
  offset(  76) =     6.46
  offset(  77) =    12.32
  offset(  78) =     7.61
  offset(  79) =   -11.58
  offset(  80) =    14.92
  offset(  81) =    -5.12
  offset(  82) = -9999.99
  offset(  83) =    13.14
  offset(  84) =   -13.39
  offset(  85) =    29.08
  offset(  86) =     9.65
  offset(  87) =     5.07
  offset(  88) =   -29.50
  offset(  89) = -9999.99
  offset(  90) =   -32.90
  offset(  91) =    -1.58
  offset(  92) =    10.27
  offset(  93) =    10.60
  offset(  94) =     6.11
  offset(  95) =   -17.37
  offset(  96) = -9999.99
  offset(  97) =   -15.29
  offset(  98) =     5.60
  offset(  99) =    18.18
  offset( 100) =     6.95
  offset( 101) =    34.63
  offset( 102) =    24.44
  offset( 103) = -9999.99
  offset( 104) =    10.31
  offset( 105) =     2.06
  offset( 106) =    24.74
  offset( 107) =    14.27
  offset( 108) =    -2.56
  offset( 109) =    -3.93
  offset( 110) = -9999.99
  offset( 111) =     8.74
  offset( 112) =    10.62
  offset( 113) =     8.13
  offset( 114) =    -9.11
  offset( 115) =    28.90
  offset( 116) =   -22.43
  offset( 117) =    15.39
  offset( 118) =     8.35
  offset( 119) = -9999.99
  offset( 120) =    23.88
  offset( 121) =    -8.30
  offset( 122) = -9999.99
  offset( 123) =   -22.34
  offset( 124) =    -3.13
  offset( 125) = -9999.99
  offset( 126) = -9999.99
  offset( 127) = -9999.99
  offset( 128) =    -5.53
  offset( 129) =    13.01
  offset( 130) =     4.93
  offset( 131) =    -6.37
  offset( 132) =   -14.43
  offset( 133) =    -1.17
  offset( 134) =   -13.50
  offset( 135) =   -29.29
  offset( 136) =     0.38
  offset( 137) =    -2.31
  offset( 138) =    17.01
  offset( 139) =   -28.94
  offset( 140) =    -9.94
  offset( 141) =    10.87
  offset( 142) =    -6.41
  offset( 143) =   -10.41
  offset( 144) =    13.21
  offset( 145) =    10.55
  offset( 146) =   -16.63
  offset( 147) =    -9.88
  offset( 148) =     3.68
  offset( 149) =   -13.44
  offset( 150) =    16.01
  offset( 151) =    -0.66
  offset( 152) =    -4.78
  offset( 153) =     3.88
  offset( 154) =     1.75
  offset( 155) =    14.87
  offset( 156) =    14.93
  offset( 157) = -9999.99
  offset( 158) =    -1.27
  offset( 159) =    11.46
  offset( 160) =     1.60
  offset( 161) =   -17.07
  offset( 162) = -9999.99
  offset( 163) =    -7.67
  offset( 164) =     6.72
  offset( 165) =   -14.06
  offset( 166) =   -13.29
  offset( 167) =    -5.26
  offset( 168) =     9.71
  offset( 169) =   -26.00
  offset( 170) =   -13.22
  offset( 171) =     1.65
  offset( 172) =   -16.16
  offset( 173) =    16.84
  offset( 174) =    10.04
  offset( 175) =   -13.46
  offset( 176) =   -11.55
  offset( 177) =    18.48
  offset( 178) =    26.41
  offset( 179) =    26.68
  offset( 180) =    52.12
  offset( 181) =    -4.79
  offset( 182) =    15.28
  offset( 183) =     7.10
  offset( 184) =    14.26
  offset( 185) =    12.19
  offset( 186) =   -11.96
  offset( 187) =    12.95
  offset( 188) =   -14.44
  offset( 189) =    -0.90
  offset( 190) =     8.53
  offset( 191) =     5.83
  offset( 192) =    14.00
  offset( 193) =    23.65
  offset( 194) =    45.50
  offset( 195) =    19.86
  offset( 196) =    20.01
  offset( 197) =    -5.01
  offset( 198) =    10.92
  offset( 199) =    10.42
  offset( 200) =   -23.83
  offset( 201) =    -8.06
  offset( 202) =    -1.51
  offset( 203) =    11.88
  offset( 204) =   -17.23
  offset( 205) =    22.72
  offset( 206) =   -15.70
  offset( 207) =    69.14
  offset( 208) =   -31.45
  offset( 209) =     4.00
  offset( 210) =     3.27
  offset( 211) =    -1.41
  offset( 212) =    17.72
  offset( 213) =    -9.82
  offset( 214) =    42.36
  offset( 215) =   -14.39
  offset( 216) =   -10.26
  offset( 217) =    16.12
  offset( 218) =   -13.05
  offset( 219) =    32.84
  offset( 220) =     8.33
  offset( 221) =   -31.72
  offset( 222) =    -6.97
  offset( 223) =     9.41
  offset( 224) =    13.02
  offset( 225) =    -4.24
  offset( 226) =     2.84
  offset( 227) =    19.86
  offset( 228) =     1.54
  offset( 229) =    -1.36
  offset( 230) =    -4.71
  offset( 231) =    -8.29
  offset( 232) =    17.26
  offset( 233) =   -27.21
  offset( 234) =     0.91
  offset( 235) =     0.88
  offset( 236) =    25.52
  offset( 237) =    -5.75
  offset( 238) =   -14.34
  offset( 239) =    -1.61
  offset( 240) =     3.03
  offset( 241) =    16.54
  offset( 242) =    -5.54
  offset( 243) =   -18.73
  offset( 244) =     9.95
  offset( 245) =     7.49
  offset( 246) =     6.72
  offset( 247) =   -10.82
  offset( 248) =    22.57
  offset( 249) =    25.84
  offset( 250) =    24.76
  offset( 251) =     9.43
  offset( 252) =    -5.13
  offset( 253) =     6.02
  offset( 254) =     1.75
  offset( 255) =    38.44
  offset( 256) = -9999.99
  offset( 257) =     4.87
  offset( 258) = -9999.99
  offset( 259) =   -22.56
  offset( 260) =     8.35
  offset( 261) =     3.63
  offset( 262) =   -21.39
  offset( 263) =    17.77
  offset( 264) =    -5.50
  offset( 265) =    -3.87
  offset( 266) =     1.14
  offset( 267) =    -1.08
  offset( 268) =     9.44
  offset( 269) =     1.99
  offset( 270) =    -5.29
  offset( 271) =    12.22
  offset( 272) =     4.93
  offset( 273) =    -0.64
  offset( 274) =    22.92
  offset( 275) =   -18.21
  offset( 276) =     0.20
  offset( 277) =    -6.97
  offset( 278) =    14.96
  offset( 279) =    14.56
  offset( 280) =    14.59
  offset( 281) =    -5.61
  offset( 282) =    15.35
  offset( 283) =   -22.48
  offset( 284) =     9.83
  offset( 285) =    19.83
  offset( 286) =   -20.28
  offset( 287) =    27.19
  offset( 288) =    11.79
  offset( 289) =    -8.16
  offset( 290) =    81.33
  offset( 291) =    -4.18
  offset( 292) =   -12.11
  offset( 293) =    -1.70
  offset( 294) =   -12.91
  offset( 295) =     1.57
  offset( 296) =   -12.75
  offset( 297) =   -23.81
  offset( 298) =     4.09
... 227 more lines ...
  524   Wed Jan 31 15:27:07 2024 TDPhotos of water flow & dew point sensor interlock box
Attachment 1 - 4 socket input connectors ( from sensor to interlock box )

Attachment 2 & 2 - 24V dc relays and internal wiring

Attachment 4 - 3 socket input connector ( from interlock box to USB-controlled ac mains relay )
Attachment 1: 20240131_160205.jpg
20240131_160205.jpg
Attachment 2: 20240131_145249.jpg
20240131_145249.jpg
Attachment 3: 20240131_145340.jpg
20240131_145340.jpg
Attachment 4: 20240131_160341.jpg
20240131_160341.jpg
  523   Mon Nov 13 11:46:54 2023 TDSummary of AIDA 24cm x 8cm 'triple' DSSSD tests
Attachment 1: Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf
Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf Summary_of_AIDA_tests_2022-3_-_Dec_2023.pdf
Attachment 2: Summary_of_AIDA_tests_2022-3_-_Dec_2023.pptx
  522   Thu Oct 12 14:01:43 2023 TDMSL type BB18 24cm x 8cm DSSSD test - update
DSSSD MSL type BB18 24cm x 8cm 3208-3/3208-21/3208-22

FEE64 configuration see https://elog.ph.ed.ac.uk/AIDA/872 attachment 2

Bias -150V -6.590uA ambient temperature +24.7 deg C d.p. +13.7 deg C RH 50.3%

BNC PB-5
amplitude 10.0V
attenuation x10
decay time 1ms
tail pulse
frequency 25Hz

PB-5 output direct to p+n junction side FEE64 aida01 or aida12, or n+n Ohmic side FEE64 aida02


aida01 1.8.L pulser peak width  61 ch FWHM ~  46keV FWHM => 5s threshold 98keV 
aida12 1.8.L pulser peak width  56 ch FWHM ~  42keV FWHM => 5s threshold 89keV
aida02 1.8.L pulser peak width 102 ch FWHM ~  77keV FWHM => 5s threshold 163keV


slow comparator  0xa 100keV ( all p+n junction FEE64s ) 
                 0xf 150keV ( all n+n Ohmic FEE64s )

per FEE64 Rate spectra - attachment 1
 p+n FEE64s ( aida010, aida01, aida09, aida12, aida03, aida11 ) rates dominated by hot channels, other channels typically <1Hz ( 25Hz pulser to aida12 )
 n+n FEE64s ( aida02, aida04 ) rates ~ 10-20Hz/channel
 Note aida06 and aida08 are not connected to anything and should be ignored


ADC data item stats - attachment 2


For further information see https://elog.ph.ed.ac.uk/AIDA/906 and https://elog.ph.ed.ac.uk/AIDA/907


To Do

- repair/replace Honeywell HSS-DPS dew point sensor
   USB-controlled ac mains relay interlock currently overrriden
   do NOT operate AIDA unattended

- aida04 asic #1 u/s - replace ASIC mezzanine

- electrically isolated test signal distribution box req'd

- aida10 asic #4
   v. high rates observed and large signal transients 
   cause unclear ... ASIC/adaptor PCB/cabling/Si wafer ?

- extended background alpha run to check all DSSSD bond wires
   pulser OFF
   slow comparator 0x64

- bPlas + 2x triple DSSSD + bPlas stack assembly and test

- all up, in beam test
Attachment 1: Screenshot_from_2023-10-12_15-13-30.png
Screenshot_from_2023-10-12_15-13-30.png
Attachment 2: Screenshot_from_2023-10-12_15-17-05.png
Screenshot_from_2023-10-12_15-17-05.png
  521   Sun Oct 1 11:47:07 2023 TDS4 cooling water
The photographs show the cooling water controls and temperature/pressure gauges outside S4 and the connections used by AIDA within S4.
Attachment 1: 1000007356.jpg
1000007356.jpg
Attachment 2: 1000007355.jpg
1000007355.jpg
  520   Thu Aug 31 15:24:56 2023 NHNew AIDA MBS PC
The AIDA MBS FDR will be x86l-119 from now on, not x86l-94

the MBS relay and startup scripts will be changed for this
  519   Mon Aug 28 12:47:56 2023 NHPower Failure 24.08.2023
There was a power failure in the morning of 24.08.2023 in the Rhein-Main area affecting GSI

The Aida workstation (aida-3) has been restarted, it is unknown if the Pis in S4 rebooted as well (there is a UPS)

29.8.23 TD Both RPi systems rebooted four days ago.
  518   Sun Jul 16 09:32:57 2023 TDFirefox browser proxy setting change
Firefox browser proxy setting changed to 'Auto-detect proxy settings for this network' ( Firefox -> Edit -> Settings -> Network Settings -> Settings )
  517   Wed Jan 18 13:40:33 2023 PJCS TDMACB settings with either Emulator or VITAR

When using the VETAR connected to the HDMI port of the root MACB the settings for all the MACB in the system 0x3.

This setting takes the Clock and Data line from the input HDMI and outputs it via teh HDMI output ports.

When using the Emulator connected via the SMA connectors on the back of the root MACB then the setting for the root MACB should be 0xD and all others should be 0x3.

Attached is the .jed file for programming the MACB and the .vhd source file to help with understanding of the settings.

Attachment 1: macb_apr20.jed
Attachment 2: macb_apr20.vhd
----------------------------------------------------------------------------------
-- Company: 
-- Engineer: 
-- 
-- Create Date:    17:03:27 03/16/2011 
-- Design Name: 
-- Module Name:    macb_top - Behavioral 
-- Project Name: 
-- Target Devices: 
-- Tool versions: 
-- Description: 
--
-- Dependencies: 
--
-- Revision: 
-- Revision 0.01 - File Created
-- Additional Comments: 
--
----------------------------------------------------------------------------------
library IEEE;
use IEEE.STD_LOGIC_1164.ALL;
use IEEE.STD_LOGIC_ARITH.ALL;
use IEEE.STD_LOGIC_UNSIGNED.ALL;

---- Uncomment the following library declaration if instantiating
---- any Xilinx primitives in this code.
library UNISIM;
use UNISIM.VComponents.all;
-- NOTE all in/out notations are relative to this unit
entity macb_apr20 is
    Port ( 
			  port1_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  port2_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  port3_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  port4_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  layer_sp : inout  STD_LOGIC_VECTOR (3 downto 0);
			  layer_trigger : out std_logic ;
           sync_return : in  STD_LOGIC_VECTOR (3 downto 1);
           selector : in  STD_LOGIC_VECTOR (3 downto 0);
           sync_select : out  STD_LOGIC_vector(1 downto 0 );
           clock200_select : out  STD_LOGIC_vector( 1 downto 0 ) ;
			  butis_divide_reset : out std_logic ;
			  butis_divide_s : out std_logic_vector( 2 downto 0 ) ;
			  clock_5 : in std_logic ;
			  sync_5 : in std_logic ;
			  trigger : in std_logic_vector( 3 downto 0 ) ;
           MBS_in : in  STD_LOGIC_VECTOR (3 downto 0);
           MBS_out : out  STD_LOGIC_VECTOR (3 downto 0));
end macb_apr20;

architecture Behavioral of macb_apr20 is
signal port1_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port1_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port1_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal port2_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port2_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port2_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal port3_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port3_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port3_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal port4_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port4_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal port4_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal layer_spi :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal layer_spo :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '0' );
signal layer_t :  STD_LOGIC_VECTOR (3 downto 0) := ( others => '1' );
signal seli : integer range 0 to 15 := 0  ;
-- well really
signal MBS_in_n : std_logic_vector( 3 downto 0 ) := "0000" ;
begin
MBS_in_n <= ( not MBS_in);
seli <= conv_integer(not selector) ;
-- MBS signal allocations to sp lines and HDMI pin. This maps to NIM connections
-- 0 :	MBS_clock10 	SP0	13
-- 1 :	MBS_reset		SP1	14
-- 2 :	MBS_reset_rq	SP2	15
-- 3 :	MBS_Trigger		SP3	16
layer_trigger <= trigger(0) or trigger(1) or trigger(2) or trigger(3) ;

-- divider controls set for pass-through
butis_divide_reset <= '1' ; -- for now don't reset ;

process ( seli , MBS_in_n, port1_spi, port2_spi, port3_spi, port4_spi, layer_spi, sync_return ,sync_5  )
-- note : & => concatenate
begin
	case seli is 
	when 0 => --- Master/ Root / MBS / Internal clock
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "00" ; -- select internal 200 MHz oscillator
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,
		
	when 1 => --- Master/ Root / MBS / BuTiS clock and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "01" ; -- select sync from external using SMA input 
		clock200_select <= "01" ; -- select external 200 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 2 => --- Master/ Branch / MBS / Next layer clock next layer SYNC
		port1_spo <= layer_spi(3) & layer_spi(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= layer_spi(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= layer_spi(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= layer_spi(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <=  '0' & '0' & port1_spi(1) & port1_spi(0) ; -- drive the clock and reset down a layer
		layer_t <= "1100" ; -- just drive the bottom two bits to the "next" port
		sync_select <= "10" ; -- select sync from next_layer 
		clock200_select <= "10" ; -- select clock from next layer
		MBS_out <=  layer_spi(3) & layer_spi(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 3 => --- Slave / Branch / MBS / Next layer clock and sync
		port1_spo <= layer_spi(3) & '0'  & layer_spi(1) & layer_spi(0);
		port1_t <= "0100" ; -- drive clock, reset, trigger only 
		port2_spo <= layer_spi(3) & '0' & layer_spi(1) & layer_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= layer_spi(3) & '0' & layer_spi(1) & layer_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= layer_spi(3) & '0' & layer_spi(1) & layer_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <=  '0' & '0' & '0' & '0' ; -- drive nothing
		layer_t <= "1111" ; -- just drive nothing down
		sync_select <= "10" ; -- select sync from next layer 
		clock200_select <= "10" ; -- select clock from next layer
		MBS_out <=  layer_spi ; -- map all the signals for monitoring ?
		butis_divide_s <= "000" ; -- s2 is 0 for pass,
		
	when 4 => --- Master/ Root / MBS / BuTiS clock / Internal SYNC / External timestamp reset
		port1_spo <= MBS_in_n(3) & '0' & MBS_in_n(1)  & '0' ;
		port1_t <= "0100" ; -- drive clock, reset, trigger only
		port2_spo <= MBS_in_n(3) & '0' & MBS_in_n(1)  & '0' ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & MBS_in_n(1)  & '0' ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & MBS_in_n(1)  & '0' ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "01" ; -- select external 50 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & MBS_in_n(1) & sync_5 ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 5 => --- Master/ Root / MBS / External 50Mhz clock / Internal Sync
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "01" ; -- select external SMA input 
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass through.
			
	when 6 => --- Master/ Root / MBS / External 100Mhz clock / Internal Sync
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "01" ; -- select external SMA input 
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "100" ; -- s2 is 1 for external, 00 for /2.
		
	when 7 => --- Fast NIM input for each FEE / Next layer clock next layer SYNC
		port1_spo <= MBS_in_n(0) & layer_spi(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(1) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(2) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <=  '0' & '0' & port1_spi(1) & port1_spi(0) ; -- drive the clock and reset down a layer
		layer_t <= "1100" ; -- just drive the bottom two bits to the "next" port
		sync_select <= "10" ; -- select sync from next_layer 
		clock200_select <= "10" ; -- select clock from next layer
		MBS_out <=  layer_spi(3) & layer_spi(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 8 => --- Fast NIM input from Input 3 for each FEE / Next layer clock next layer SYNC
		port1_spo <= MBS_in_n(3) & layer_spi(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <=  '0' & '0' & port1_spi(1) & port1_spi(0) ; -- drive the clock and reset down a layer
		layer_t <= "1100" ; -- just drive the bottom two bits to the "next" port
		sync_select <= "10" ; -- select sync from next_layer 
		clock200_select <= "10" ; -- select clock from next layer
		MBS_out <=  layer_spi(3) & layer_spi(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,
	
	when 9 => --- Master/ Root / Internal clock / sync_returns to NIM
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "00" ; -- select sync from port 1 
		clock200_select <= "00" ; -- select internal 200 MHz oscillator
		MBS_out <=  sync_return(3) & sync_return(2) & sync_return(1) & '0' ;
		butis_divide_s <= "000" ; -- s2 is 0 for pass,

	when 10 => --- Master/ Root / MBS / BuTiS clock /2 and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= "0000" ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "01" ; -- select sync from external using SMA input 
		clock200_select <= "01" ; -- select external 200 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n ; -- for testing NIM I/O
		butis_divide_s <= "100" ; --  s2 = 1 and s1,s0 decode to 00=>/2 , 01=>/4, /8 , /16

	when 12 => --- Master/ Root / MBS / BuTiS clock /2 and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "01" ; -- select sync from external using SMA input 
		clock200_select <= "01" ; -- select external 200 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "100" ; --  s2 = 1 and s1,s0 decode to 00=>/2 , 01=>/4, /8 , /16

	when 13 => --- Master/ Root / MBS / BuTiS clock /4 and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port3_t <= "0100" ; -- drive clock, reset, trigger only
		port4_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port4_t <= "0100" ; -- drive clock, reset, trigger only
		layer_spo <= ( others => '0' ) ;
		layer_t <= ( others => '1' ) ; -- disable the drive to the "next" port
		sync_select <= "01" ; -- select sync from external using SMA input 
		clock200_select <= "01" ; -- select external 200 MHz oscillator using SMA input
		MBS_out <=  MBS_in_n(3) & MBS_in_n(2) & port1_spi(1) & port1_spi(0) ;
		butis_divide_s <= "101" ; --  s2 = 1 and s1,s0 decode to  01=>/4
		
	when 14 => --- Master/ Root / MBS / BuTiS clock /8 and SYNC
		port1_spo <= MBS_in_n(3) & MBS_in_n(2) & '0' & '0' ;
		port1_t <= "0011" ; -- drive trigger and reset request only
		port2_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
		port2_t <= "0100" ; -- drive clock, reset, trigger only 
		port3_spo <= MBS_in_n(3) & '0' & port1_spi(1) & port1_spi(0) ;
... 161 more lines ...
Attachment 3: zybo.jpg
zybo.jpg
Attachment 4: MACB.jpg
MACB.jpg
  516   Fri Dec 16 14:02:12 2022 NHAIDA System off for christmas break
The AIDA NIM crate, pis and workstation have been powered off for the Christmas break and will not be accessible
  515   Thu Oct 6 16:51:49 2022 NHOscilloscope analysis
Investigating AIDA noise with a TA041 differential probe and oscilloscope 

AC Mains (DESPEC platform AC, L-N)
Probe attentuation = 1:100

Fig 1: Main AC waveform [X: 5ms/div, Y: 100 V/div]
Fig 2: Zoomed in at peak (20 V FSR, any less and the waveform clipped) [X: 10us/div, Y: 20V/div]
Fig 3: Longer time base and FFT of 0-5 MHz. No significant frequency harmonics noticed [X: 5ms/div, Y:20 V, FFT X: 500 kHz/div, Y: 10 dBm/div]

No significant noise or distortion present, fully within any AC specification. 
Note that at the moment there is almost no load on AC
Equipment on on DESPEC rack: AIDA NIM, AIDA Raspberry Pis, bPlas PC (+ WR) + 2x DESPEC NIM crates
 No autofill, VME crate or detectors
All big machines at GSI (SIS, FRS) off (suspect pumps are on)

Ion catcher not on (I think under repair)

-

FEE PSU studies
Probe connected to 5V exposed power pin on FEE64 (+v) and to grounding crimp on FEE64 (-v)
No adapter board connected
Attentuation = 1:10


Fig 4: FFT when FEEs are *off* - essentially probe+scope noise [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]
Fig 5: FFT when FEEs are *on* - notice 1.4 MHz peak in FFT, also seen on ADC waveform readout before (fig 6) [X: 5ms/div, Y: 100 mV/div, FFT: X: 500 kHz/div, Y: 10 dBm/div]

Fig 7: 500 ns/div 5V output on FEE, single FEE on the PSU [X: 500 ns/div, Y: 100 mV/div]

Note average max voltage is 5.31 V (power on) and ~ 70 mV "peak to peak" -might be from probe/scope?

Also see voltage changes with FEE power draw:

Power on  : 5.45 V (different scale to above)
SETUP ran : 5.51 V 
FADCs off : 5.86 V
ACQ Go:   : Unchanged; ASIC threshold 0xa: Unchanged

Also check situation on a fully loaded PSU (8 fees connected and powered on)

Power on:   5.29 V (fig 8)
SETUP ran:  5.36 V (fig 9)
FADCs off:  5.64 V (fig 10)

All X: 500 ns/div, Y: 100 mV/div

Both cases observe voltage rises as current draw drops (as expected for voltage drop along a cable)

Noise on 'scope seems to get slightly worse with reduced current (and higher voltage)

No sign of strong 100 kHz noise as seen in ADC traces beforehand

Todo:

- Check -6V and 7V rails
- Check 5V and noise when front-end card is added and pulser/HV connected
- Check between two FEE64 grounds 
- Check direct out of PSU vs ground to see if 1.4 MHz appears on PSU side or FEE64 side

-
11.10.22 Updates

Attachement 11 - 5V PSU on upper PSU with no FEEs attached whatsoever. No 1.4 MHz (on FFT) but clear low frequency beats from switching - presumably low/no load behaviour

Attachement 12 - 5V PSU on aida12 with 8 FEEs on PSU. Longer time base to allow lower frequencies in FFT. 1.4 MHz switching spikes visible but nothing around 100 kHz region

Attachments 13-16: 5V PSU on aida12 at 20 mV/div vertical and 1, 0.5, 2, 5 us/div horizontal respectively

12.10.22 Updates

Attachment 17: -6V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: -6.21 V

Attachment 19: 7V PSU on aida12 with 8 FEEs on PSU. 2 us/div timebase. 20 mV/div amplitude
Attachment 18: 10 ms timebase and FFT
Average voltage: 7.46V

Measurement between AIDA12 ground and Reference ground/copper bar
+ve (red probe) attached to copper bar at ground point (not strong connection at present)
-ve (black probe) attached to ground crimp on aida12 (connected to cooling plate)
aida12 no adapter board connected: connections are PSU, Ethernet, HDMI and TTY only

Attachment 21: 5 us/div 100 mv/div waveform, big oscillations present. Not seen before FEEs turned on (8 FEES, 1-7+12)
Attachment 22: 10 ms/div for FFT, sharp peak at exactly 100 kHz observed...


Attachment 23: Between 5V PSU (+ve) and 19" rack (-ve) with  no FEEs connected to PSU
 See strong 100 kHz oscillations too, note that voltage isn't 5V as PSU is floating w.r.t. ground
 Looks to be common mode noise (on both 5V and Return of PSU)

Attachment 24: Same as 21 but using thick crocodile clips on probe to ground and aida12. Noise is attenuated but still present
Attachment 1: SCRN0086.PNG
SCRN0086.PNG
Attachment 2: SCRN0090.jpg
SCRN0090.jpg
Attachment 3: SCRN0113.jpg
SCRN0113.jpg
Attachment 4: SCRN0107.jpg
SCRN0107.jpg
Attachment 5: SCRN0108.jpg
SCRN0108.jpg
Attachment 6: Image_Pasted_at_2022-10-6_15-51.jpg
Image_Pasted_at_2022-10-6_15-51.jpg
Attachment 7: SCRN0098.jpg
SCRN0098.jpg
Attachment 8: SCRN0109.jpg
SCRN0109.jpg
Attachment 9: SCRN0110.jpg
SCRN0110.jpg
Attachment 10: SCRN0111.jpg
SCRN0111.jpg
Attachment 11: SCRN0115.PNG
SCRN0115.PNG
Attachment 12: SCRN0121.PNG
SCRN0121.PNG
Attachment 13: SCRN0124.PNG
SCRN0124.PNG
Attachment 14: SCRN0125.PNG
SCRN0125.PNG
Attachment 15: SCRN0126.PNG
SCRN0126.PNG
Attachment 16: SCRN0127.PNG
SCRN0127.PNG
Attachment 17: SCRN0132.PNG
SCRN0132.PNG
Attachment 18: SCRN0129.PNG
SCRN0129.PNG
Attachment 19: SCRN0139.PNG
SCRN0139.PNG
Attachment 20: SCRN0136.PNG
SCRN0136.PNG
Attachment 21: SCRN0141.PNG
SCRN0141.PNG
Attachment 22: SCRN0143.PNG
SCRN0143.PNG
Attachment 23: SCRN0149.PNG
SCRN0149.PNG
Attachment 24: SCRN0153.PNG
SCRN0153.PNG
  514   Wed Sep 14 19:07:07 2022 PJCSINFO : Three Merger Statistics explained

There are three Merger statistics that can be used to better understand how the data flow through the Merger system is proceeding.

Two are from the Link task and one is from the Merger.They are all "No data buffers avaliable" with #1, #2, #3 at the end.

#1: This is incremented when the Link task has a data item to put in the queue for the Merge process but there is no room.

#2: This is incremented when the Link task has found no room in the queue for the Merge process ( #1 ) , waited , tried again and failed. 

#3: This the other end of the queue. When the Merge task requests a data item from a Link task queue and there is nothing available.

 

  513   Thu Sep 8 12:37:18 2022 NHProxy Port Changed
The proxy in Firefox, Yum and AnyDesk has been changed as the old wasn't working

proxy.gsi.de port 3128 is now in use
  512   Thu Sep 8 12:31:25 2022 NHRetrying AIDA DataAcq v10
Startup AIDA with ribbon cable connected to aida03 and aida07 for noise

Setup and run with waveforms enabled. Discriminators ADC power etc as default

Try to push above 200k as this is where we saw issues before... lowering threshold to 0x3 pushes rates to

aida03 - 320k
aida07 - 254k

Startup merger and observe rates

aida03 - 224k
aida07 - 213k

Rate drop observed as before.

Now update aidacommon to point to AidaExecV10 and powercycle FEEs

Rates again with 0x3 

aida03 - 315k
aida07 - 252k

Restart with data transfer ON

aida03 - 317k
aida07 - 262k

No errors in merger terminal or "Merge time errors" statistic

Will keep running 
Attachment 1: AnyDeskMSI_2022-09-08_13-35-50.png
AnyDeskMSI_2022-09-08_13-35-50.png
Attachment 2: AnyDeskMSI_2022-09-08_13-35-58.png
AnyDeskMSI_2022-09-08_13-35-58.png
Attachment 3: AnyDeskMSI_2022-09-08_13-36-02.png
AnyDeskMSI_2022-09-08_13-36-02.png
  511   Tue Aug 30 13:48:32 2022 NHAIDA Single Switch Configuration
The second switch was moved back to CARME so AIDA has been configured back to using a single switch

aida02/aida04/aida06/aida08 updated back to first switch as per https://elog.ph.ed.ac.uk/DESPEC/433

Additionally a ribbon cable is attached to aida01 and aida05 to introduce some noise into the system
  510   Tue Jul 5 08:53:20 2022 TDTo Do
In no particular order

1) CAEN 83xx series NIM bin (Ortec 533A output noise issue)

   observe +/- 6V, 12V, 24V lines with/without load 

   try new CAEN NIM bin and/or NIM bin of different type

2) Measure actual voltages at FEE64 power connector input

   OH suggests fab of power adaptor for safe observation - contact EW

3) rev B adaptor PCB

   invert 125 way ERNI - check for mech conflicts
   paired HV input (avoid Lemo-00 T pieces)
   consider isolating test/HV Lemo-00 shells from PCB ground (loop elimination)
   straight jumpers
   shrouded Samtec headers - consider mech issues/consequences of using eject clips too
   re-visit HV filtering & separate trace ground

4) isolation transformer

   as practical matter may be necessary to operate all platform from isolation transformer
   consider hire of appropriate unit

   need method to measure isolation - will require permit to work or equiv

5) investigate S4 area ac mains

   NH discussing with GSI electricians

6) Systematic measurement of AIDA PSU noise

   Spec linear AIDA FEE64 PSU

7) Redesign of snout
   Return to 1mm welded box Al for lower stage of snout  for added rigidity

8) Revisit calculation of cable lengths. Particularly for the triple

9) 



   
  509   Wed Jun 29 10:48:26 2022 NH, OHAIDA Dismounted
All detectors removed from single and triple AIDA snouts

Empty snouts *and* DSSDs (in boxes) stored in NH office 
  508   Tue Jun 28 10:11:35 2022 OH, NHMIDAS Data Aq V10
11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
      When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
      i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
      This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics

      Patrick has produced V10 which removes these.

      When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
      WR items for 0xE - attachment 1
      WR items for 0x7 - attachment 2

      However we see in the NewMerger terminal the message shown in attachment 3 frequently.
      Also we note that the merger time error counter is also going up.
      Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
      Are we having dead time issues which is causing time warps?

      Does each buffer from the MIDAS Data Acq start with a full WR timestamp?

      aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way?
Attachment 1: 220628_1117_Rollover0xE.png
220628_1117_Rollover0xE.png
Attachment 2: 220628_1119_Merger_Stats_0x7.png
220628_1119_Merger_Stats_0x7.png
Attachment 3: NewMerger_Dump.txt
MERGE Data Link (17671): block sequence error: ID 2, expected 462; received 0
MERGE Data Link (17669): block sequence error: ID 0, expected 461; received 0
MERGE Data Link (17672): block sequence error: ID 3, expected 462; received 0
MERGE Data Link (17670): block sequence error: ID 1, expected 462; received 0
MERGE Data Link (17674): block sequence error: ID 5, expected 463; received 0
MERGE Data Link (17676): block sequence error: ID 7, expected 462; received 0
MERGE Data Link (17675): block sequence error: ID 6, expected 461; received 0
MERGE Data Link (17673): block sequence error: ID 4, expected 462; received 0
MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

MERGE Actor (17699): Working with 0 from 8 data sources

  507   Tue Jun 28 09:37:19 2022 NHTues 28 June 08:00-
Experiment over

10:37 - Stop DAQ & Tape 

S4 enters controlled access and they uncable bPlas

Will dismount AIDA snout after
  506   Mon Jun 27 23:11:47 2022 TDTuesday 28 June 00:00-08:00
00:07 Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_896 - attachment 1
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 2 & 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6 & 7

00:15 Check ASIC control all FEE64s, all ASICs


02:03

analysis of file S505/R5_914 - attachment 8
 zero timewarps
 deadtime all FEE64s << 1% 

per FEE64 1.8.H spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 94 ch FWHM

per FEE64 stat & rate spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


03:31 S505 PI Anabel declares experiment end - following periods of beam loss and FRS DAQ issues today


03:41

analysis of file S505/R5_926 - attachment 19
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24


07:00

analysis of file S505/R5_954 - attachment 25
 zero timewarps
 deadtime all FEE64s << 1% 

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30
Attachment 1: R5_896
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261821326 (  595374.9 Hz)
        Other data format:      98676 (     224.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        406 (       0.9 Hz)
                               RESUME:        406 (       0.9 Hz)
                              SYNC100:      33638 (      76.5 Hz)
                              WR48-63:      33638 (      76.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      30588 (      69.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      439.759 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                1.599                0.000
  2                0.000                0.000
  3                0.043                0.000
  4                0.000              153.964
  5                0.000                0.000
  6                0.053                0.000
  7                0.015                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   19090298      35782          0          0          0          0       2597       2597          0      30588          0          0
  1   70343282      19202          0          0        400        400       9201       9201          0          0          0          0
  2   10361360       2782          0          0          0          0       1391       1391          0          0          0          0
  3   39578798      10102          0          0          3          3       5048       5048          0          0          0          0
  4    3065688        742          0          0          0          0        371        371          0          0          0          0
  5   15482560       3946          0          0          0          0       1973       1973          0          0          0          0
  6   51969876      12870          0          0          2          2       6433       6433          0          0          0          0
  7   51929464      13250          0          0          1          1       6624       6624          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.371s (  904.697 blocks/s,  56.544 Mb/s)
Attachment 2: Screenshot_from_2022-06-28_00-06-58.png
Screenshot_from_2022-06-28_00-06-58.png
Attachment 3: Screenshot_from_2022-06-28_00-06-35.png
Screenshot_from_2022-06-28_00-06-35.png
Attachment 4: Screenshot_from_2022-06-28_00-06-00.png
Screenshot_from_2022-06-28_00-06-00.png
Attachment 5: Screenshot_from_2022-06-28_00-05-30.png
Screenshot_from_2022-06-28_00-05-30.png
Attachment 6: Screenshot_from_2022-06-28_00-04-42.png
Screenshot_from_2022-06-28_00-04-42.png
Attachment 7: Screenshot_from_2022-06-28_00-03-56.png
Screenshot_from_2022-06-28_00-03-56.png
Attachment 8: R5_914
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261460369 (  646962.6 Hz)
        Other data format:     459631 (    1137.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        616 (       1.5 Hz)
                               RESUME:        616 (       1.5 Hz)
                              SYNC100:      33505 (      82.9 Hz)
                              WR48-63:      33505 (      82.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     391389 (     968.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     131720 (     325.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      404.135 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.615                0.000
  1                2.119                0.000
  2                0.420                0.000
  3                0.323                0.000
  4                0.124              111.840
  5                0.000                0.000
  6                0.696                0.000
  7                0.001                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   19284409      33077          0          0          6          6       2512       2512          0      28041          0      31815
  1   69379431      18886          0          0        391        391       9052       9052          0          0          0      26209
  2   20984122      96945          0          0          6          6       2775       2775          0      91383          0      19775
  3   56761004     105837          0          0        202        202       7127       7127          0      91179          0       9889
  4    3261224        838          0          0          2          2        417        417          0          0          0       9818
  5   13707273       3434          0          0          0          0       1717       1717          0          0          0      11918
  6   42321326     100064          0          0          8          8       5351       5351          0      89346          0      17237
  7   35761580     100550          0          0          1          1       4554       4554          0      91440          0       5059
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.648s (  897.667 blocks/s,  56.104 Mb/s)
Attachment 9: Screenshot_from_2022-06-28_02-14-50.png
Screenshot_from_2022-06-28_02-14-50.png
Attachment 10: Screenshot_from_2022-06-28_02-13-46.png
Screenshot_from_2022-06-28_02-13-46.png
Attachment 11: Screenshot_from_2022-06-28_02-12-22.png
Screenshot_from_2022-06-28_02-12-22.png
Attachment 12: Screenshot_from_2022-06-28_02-11-10.png
Screenshot_from_2022-06-28_02-11-10.png
Attachment 13: Screenshot_from_2022-06-28_02-10-07.png
Screenshot_from_2022-06-28_02-10-07.png
Attachment 14: Screenshot_from_2022-06-28_02-08-49.png
Screenshot_from_2022-06-28_02-08-49.png
Attachment 15: Screenshot_from_2022-06-28_02-05-26.png
Screenshot_from_2022-06-28_02-05-26.png
Attachment 16: Screenshot_from_2022-06-28_02-04-36.png
Screenshot_from_2022-06-28_02-04-36.png
Attachment 17: Screenshot_from_2022-06-28_02-04-11.png
Screenshot_from_2022-06-28_02-04-11.png
Attachment 18: Screenshot_from_2022-06-28_02-03-39.png
Screenshot_from_2022-06-28_02-03-39.png
Attachment 19: R5_926
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261820610 (  589706.0 Hz)
        Other data format:      99390 (     223.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        624 (       1.4 Hz)
                               RESUME:        624 (       1.4 Hz)
                              SYNC100:      33654 (      75.8 Hz)
                              WR48-63:      33654 (      75.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      30834 (      69.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      443.985 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                1.081                0.000
  2                0.000                0.000
  3                0.281                0.000
  4                0.000              152.429
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   15622398      34978          0          0          0          0       2072       2072          0      30834          0          0
  1   68236330      18574          0          0        381        381       8906       8906          0          0          0          0
  2   18744892       4984          0          0          0          0       2492       2492          0          0          0          0
  3   59875268      15514          0          0        243        243       7514       7514          0          0          0          0
  4    2759883        664          0          0          0          0        332        332          0          0          0          0
  5   13541412       3478          0          0          0          0       1739       1739          0          0          0          0
  6   45470554      11726          0          0          0          0       5863       5863          0          0          0          0
  7   37569873       9472          0          0          0          0       4736       4736          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.045s (  913.114 blocks/s,  57.070 Mb/s)
Attachment 20: Screenshot_from_2022-06-28_03-44-20.png
Screenshot_from_2022-06-28_03-44-20.png
Attachment 21: Screenshot_from_2022-06-28_03-44-00.png
Screenshot_from_2022-06-28_03-44-00.png
Attachment 22: Screenshot_from_2022-06-28_03-43-05.png
Screenshot_from_2022-06-28_03-43-05.png
Attachment 23: Screenshot_from_2022-06-28_03-41-50.png
Screenshot_from_2022-06-28_03-41-50.png
Attachment 24: Screenshot_from_2022-06-28_03-41-21.png
Screenshot_from_2022-06-28_03-41-21.png
Attachment 25: R5_954
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261819650 (  564425.6 Hz)
        Other data format:     100350 (     216.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        346 (       0.7 Hz)
                               RESUME:        346 (       0.7 Hz)
                              SYNC100:      33728 (      72.7 Hz)
                              WR48-63:      33728 (      72.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      32202 (      69.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      463.869 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.860                0.000
  2                0.000               16.297
  3                0.000                0.000
  4                0.000              130.996
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   19510795      37524          0          0          0          0       2661       2661          0      32202          0          0
  1   71776267      19370          0          0        343        343       9342       9342          0          0          0          0
  2    7596519       1752          0          0          0          0        876        876          0          0          0          0
  3   39174917      10150          0          0          0          0       5075       5075          0          0          0          0
  4    3312486        838          0          0          0          0        419        419          0          0          0          0
  5   15419570       4004          0          0          0          0       2002       2002          0          0          0          0
  6   56857618      14472          0          0          3          3       7233       7233          0          0          0          0
  7   48171478      12240          0          0          0          0       6120       6120          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.318s (  906.044 blocks/s,  56.628 Mb/s)
Attachment 26: Screenshot_from_2022-06-28_07-03-01.png
Screenshot_from_2022-06-28_07-03-01.png
Attachment 27: Screenshot_from_2022-06-28_07-02-34.png
Screenshot_from_2022-06-28_07-02-34.png
Attachment 28: Screenshot_from_2022-06-28_07-01-31.png
Screenshot_from_2022-06-28_07-01-31.png
Attachment 29: Screenshot_from_2022-06-28_07-01-03.png
Screenshot_from_2022-06-28_07-01-03.png
Attachment 30: Screenshot_from_2022-06-28_07-00-31.png
Screenshot_from_2022-06-28_07-00-31.png
  505   Mon Jun 27 17:03:09 2022 MAMonday 27th June 16:00-00:00

16:00 Took over the shift from OH no beam yet.

18:00 Still no beam yet.

Statistics, Temperature, Current are checked and attached 1-3

system wide checks same as last updated in the previoues shift.

22:00 The beam is back but not taking data yet! FRS team doing some checkings

Statistics, Temperature, Current are checked and attached 4-6

system wide checks same as last updated in the previoues shift.

23:30 beam is back and taking data

Attachment 1: Statistics2022-06-27_18-02-25.png
Statistics2022-06-27_18-02-25.png
Attachment 2: Temperature2022-06-27_18-01-14.png
Temperature2022-06-27_18-01-14.png
Attachment 3: Current2022-06-27_17-59-55.png
Current2022-06-27_17-59-55.png
Attachment 4: Statics2022-06-27_22-12-07.png
Statics2022-06-27_22-12-07.png
Attachment 5: Temperature2022-06-27_22-11-21.png
Temperature2022-06-27_22-11-21.png
Attachment 6: Current2022-06-27_22-10-12.png
Current2022-06-27_22-10-12.png
  504   Mon Jun 27 06:45:22 2022 OHMonday 27th June 08:00-16:00
07:45 Spoke to David and the beam has been gone since about 05:30
      Reason for the loss of beam is a vacuum issue before the FRS
      They are waiting for the experts

08:31 Statistics ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage currrents ok - attachment 3
      ASIC clock check ok
	
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2
	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  

      Currently on file R5_771
      Analysis of file R5_770 (No beam) - attachment 4
      Around 0.25% deadtime on AIDA02 rest even less

09:02 Current free HDD space 965 GB
      Current tape server rate 4979 kB/s
      Free space taking data rate at 5700 kB/s (Closer to beam value) 47 hours

11:41 Statistics ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7
      ASIC clock check ok
	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5cf : 	 146  
      aida08 fault 	 0xf1be : 	 0xf2ba : 	 252  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x41 : 	 23  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Note that there has been no change in the White Rabbit errors or FPGA faults since very early morning.
      Could the rate of accrual in errors be proportional to the data rate. Faster data rate, errors occur more frequently?
Attachment 1: 220626_0830_Stats.png
220626_0830_Stats.png
Attachment 2: 220626_0830_Temp.png
220626_0830_Temp.png
Attachment 3: 220626_0831_Bias.png
220626_0831_Bias.png
Attachment 4: R5_770_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261823568 (  639881.5 Hz)
        Other data format:      96432 (     235.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        451 (       1.1 Hz)
                               RESUME:        451 (       1.1 Hz)
                              SYNC100:      33524 (      81.9 Hz)
                              WR48-63:      33524 (      81.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      28482 (      69.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      409.175 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.033                0.000
  1                1.307                0.000
  2                0.000                0.000
  3                0.206                0.000
  4                0.000              157.882
  5                0.000                0.000
  6                0.000                0.000
  7                0.139                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   20319054      33950          0          0          1          1       2733       2733          0      28482          0          0
  1   68104715      18442          0          0        318        318       8903       8903          0          0          0          0
  2   24150534       6320          0          0          0          0       3160       3160          0          0          0          0
  3   55928626      14410          0          0        131        131       7074       7074          0          0          0          0
  4    2503858        618          0          0          0          0        309        309          0          0          0          0
  5   12980974       3340          0          0          0          0       1670       1670          0          0          0          0
  6   41271038      10344          0          0          0          0       5172       5172          0          0          0          0
  7   36564769       9008          0          0          1          1       4503       4503          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.158s (  777.488 blocks/s,  48.593 Mb/s)
  503   Sun Jun 26 23:04:30 2022 Marcnew shift - Monday 27 June 0:00 to 8:00

0:00 Experiment continues to run smoothly. AIDA DAQ has been rather stable. Leakage current has gone up for the past two days in both DSSDs (combination of high-Z and high-intensity beam and external high temperature).

Stats & Temperatures (VIRTEX,PSU, ASICs)  all ok.

At 0:30

Stats ok - Attachment 1

Temp ok - Attachement 2

HV-LC -Attachment 3

At 2:20

Stats ok - Attachment 4

Temp ok - Attachement 5

HV-LC -Attachment 6

Wide Checks:

Clock status test result: Passed 8, Failed 0  

    Understand status as follows
    Status bit 3 : firmware PLL that creates clocks from external clock not locked 
    Status bit 2 : always logic '1'
    Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
    Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
    If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration (same as before):

    FEE64 module aida01 failed
    FEE64 module aida02 failed
    FEE64 module aida03 failed
    FEE64 module aida04 failed
    FEE64 module aida05 failed
    FEE64 module aida06 failed  
    FEE64 module aida07 failed
    FEE64 module aida08 failed
    Calibration test result: Passed 0, Failed 8

    If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

WR decoder status:

         Base         Current     Difference
aida07 fault      0xc53d :      0xc5c9 :      140  
aida08 fault      0xf1be :      0xf2b2 :      244  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp check:
             Base         Current         Difference
aida07 fault      0x2a :      0x41 :      23  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

At 4:15:

Stats ok - Attachment 7

Temp ok - Attachement 8

HV-LC -Attachment 9

Wide Checks: No change

At 7:15: (no beam since ~6am -> background run)

Stats ok - Attachment 10

Temp ok - Attachement 11

HV-LC -Attachment 12

Wide Checks: No change

Attachment 1: Stats-Screenshot_from_2022-06-27_00-30-58.png
Stats-Screenshot_from_2022-06-27_00-30-58.png
Attachment 2: Temp-Screenshot_from_2022-06-27_00-31-44.png
Temp-Screenshot_from_2022-06-27_00-31-44.png
Attachment 3: HV-LC-Screenshot_from_2022-06-27_00-30-18.png
HV-LC-Screenshot_from_2022-06-27_00-30-18.png
Attachment 4: Stats-Screenshot_from_2022-06-27_02-22-34.png
Stats-Screenshot_from_2022-06-27_02-22-34.png
Attachment 5: Temp-Screenshot_from_2022-06-27_02-19-42.png
Temp-Screenshot_from_2022-06-27_02-19-42.png
Attachment 6: HV-LC-Screenshot_from_2022-06-27_02-19-05.png
HV-LC-Screenshot_from_2022-06-27_02-19-05.png
Attachment 7: Stats-Screenshot_from_2022-06-27_04-17-22.png
Stats-Screenshot_from_2022-06-27_04-17-22.png
Attachment 8: Temp-Screenshot_from_2022-06-27_04-16-40.png
Temp-Screenshot_from_2022-06-27_04-16-40.png
Attachment 9: HV-LC-Screenshot_from_2022-06-27_04-14-57.png
HV-LC-Screenshot_from_2022-06-27_04-14-57.png
Attachment 10: Stats-Screenshot_from_2022-06-27_07-15-17.png
Stats-Screenshot_from_2022-06-27_07-15-17.png
Attachment 11: Temp-Screenshot_from_2022-06-27_07-14-02.png
Temp-Screenshot_from_2022-06-27_07-14-02.png
Attachment 12: HV-LC-Screenshot_from_2022-06-27_07-13-09.png
HV-LC-Screenshot_from_2022-06-27_07-13-09.png
  502   Sun Jun 26 08:51:20 2022 OH, NHSunday 26 June 08:00-24:00
09:51 Taken over from Tom following the night shift
      Experiment is still running smoothly
      Compression of the files is complete up to the start of R5.
      Have started compression of files in R5 which should run up to R5_499
      Currently on R5_528

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      System wide checks:
          		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc594 : 	 87  
      aida08 fault 	 0xf1be : 	 0xf271 : 	 179  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 	Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Current merger rate is 2E6-4E6 events per second
      Current tapeserver rate is 5800 kB/s
      Free HDD space is 1.1 TB
      At current rates will last 54 hours which should see out the experiment
      Experiment currently scheduled to finish at 6am on Tuesday morning (May get until 8am)
      
      Analysis of R5_528 - attachment 4
      Deadtime of AIDA02 currently around 15%

11:56 Statistics ok - attachment 5
      Temps ok - attachment 6
      Bias and leakage currents ok - attachment 7
      System wide checks:
      Clocks all ok 
      
         		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59c : 	 95  
      aida08 fault 	 0xf1be : 	 0xf27a : 	 188  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3b : 	 17  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of file R5_458 - attachment 8
      Deadtime in AIDA02 only 11.5% in this file
      TapeServer rate still 5.5 MB/s

14:36 Has been no beam for the last while or so.
      With no beam AIDA02 has 0.25% deadtime so the deadtime is almost entirely due to the spill on time
      Stats- attachment 9
      Temp - attachment 10
      Bias and leakage currents ok - attachment 11
      System wide checks:
      Clock ok
      	
        		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc59d : 	 96  
      aida08 fault 	 0xf1be : 	 0xf27c : 	 190  
      White Rabbit error counter test result: Passed 6, Failed 2
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3c : 	 18  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

13:30 Beam taken to change an ion source

14:57 Beam back

16:21 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14

      System wide checks - ASIC clocks ok 
	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5a3 : 	 102  
      aida08 fault 	 0xf1be : 	 0xf285 : 	 199  
      White Rabbit error counter test result: Passed 6, Failed 2

	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3e : 	 20  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Currently on file R5_592
      Analysis of R5_591 - Attachment 15
      Deadtime of AIDA02 sitting at 17%

[NH taking over for OH so he can get home]

18:53 - FRS DAQ problems mean we weren't taking DESPEC data for the past hour or so... now all back
        Statitics ok - attachement 16
        Temps ok - attachement 17
        Bias & leakage ok - attachement 18

        System wide checks - 
        Clocks OK
        ADC Calibration N/A 
	WR Decoder -
        		 Base 		Current 	Difference
        aida07 fault 	 0xc53d : 	 0xc5ac : 	 111  
        aida08 fault 	 0xf1be : 	 0xf28c : 	 206  
        White Rabbit error counter test result: Passed 6, Failed 2
        FPGA -
         Base 		Current 		Difference
        aida07 fault 	 0x2a : 	 0x3f : 	 21  
        PLL OK

        Currently on file R5_621
        Analysis of R5_620 - Attachement 19
        aida02 deadtime only 5%, all others negligible 

20:20 Stats ok - attachment 20
      Temps ok - attachment 21
      Bias and leakage currents ok - attachment 22
      System wide checks:
      Clock ok
    	
	        	 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ad : 	 112  
      aida08 fault 	 0xf1be : 	 0xf293 : 	 213  
      White Rabbit error counter test result: Passed 6, Failed 2

     	
			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x3f : 	 21  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R6_637 - attachment 23

22:13 Statistics ok - attachment 24
      Temperatures ok - attachment 25
      Bias and leakage currents ok - attachment 26
      ASIC clock check ok

	
		         Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc5ba : 	 125  
      aida08 fault 	 0xf1be : 	 0xf2a2 : 	 228  
      White Rabbit error counter test result: Passed 6, Failed 2

			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x40 : 	 22  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_658 - attachment 27
Attachment 1: 220626_0949_stats.png
220626_0949_stats.png
Attachment 2: 220626_0950_Temps.png
220626_0950_Temps.png
Attachment 3: 220626_0951_Bias.png
220626_0951_Bias.png
Attachment 4: R5_528_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260639000 (  776334.1 Hz)
        Other data format:    1281002 (    3815.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1133 (       3.4 Hz)
                               RESUME:       1133 (       3.4 Hz)
                              SYNC100:      33251 (      99.0 Hz)
                              WR48-63:      33251 (      99.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1212234 (    3610.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     393457 (    1171.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      335.730 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                8.363                0.000
  1               44.893                0.000
  2                0.179                0.000
  3                3.779                0.000
  4                0.000               44.522
  5                0.034                0.000
  6                0.166                0.000
  7                0.017                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31317521      31305          0          0        207        207       4035       4035          0      22821          0      98396
  1   68454688      19188          0          0        693        693       8901       8901          0          0          0      59818
  2   19369571     305849          0          0         35         35       2468       2468          0     300843          0      63063
  3   45019317     306201          0          0        130        130       5636       5636          0     294669          0      32623
  4    4769649       1252          0          0          0          0        626        626          0          0          0      31000
  5   17853070       4600          0          0         15         15       2285       2285          0          0          0      37328
  6   42448852     303192          0          0         44         44       5356       5356          0     292392          0      54344
  7   31406332     309415          0          0          9          9       3944       3944          0     301509          0      16885
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.891s (  763.894 blocks/s,  47.743 Mb/s)
Attachment 5: 220626_1155_stats.png
220626_1155_stats.png
Attachment 6: 220626_1155_Temp.png
220626_1155_Temp.png
Attachment 7: 220626_1156_Bias.png
220626_1156_Bias.png
Attachment 8: R5_548_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260758427 (  740797.4 Hz)
        Other data format:    1161573 (    3300.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1084 (       3.1 Hz)
                               RESUME:       1084 (       3.1 Hz)
                              SYNC100:      33311 (      94.6 Hz)
                              WR48-63:      33311 (      94.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1092783 (    3104.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     356241 (    1012.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      351.997 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                5.340                0.000
  1               40.660                0.000
  2                0.175                0.000
  3                2.385                0.000
  4                0.000               67.319
  5                0.032                0.000
  6                0.142                0.000
  7                0.019                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   29258900      32253          0          0        166        166       3875       3875          0      24171          0      88948
  1   70599060      19658          0          0        716        716       9113       9113          0          0          0      56350
  2   19313501     275533          0          0         28         28       2458       2458          0     270561          0      56248
  3   44887469     278330          0          0        112        112       5682       5682          0     266742          0      29387
  4    4430708       1118          0          0          0          0        559        559          0          0          0      27825
  5   17540711       4456          0          0         11         11       2217       2217          0          0          0      33663
  6   43048879     271297          0          0         40         40       5377       5377          0     260463          0      48745
  7   31679199     278928          0          0         11         11       4030       4030          0     270846          0      15075
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   40.309s (  793.875 blocks/s,  49.617 Mb/s)
Attachment 9: 220626_1435_stats.png
220626_1435_stats.png
Attachment 10: 220626_1435_Temp.png
220626_1435_Temp.png
Attachment 11: 220626_1436_Bias.png
220626_1436_Bias.png
Attachment 12: 220626_1620_Stats.png
220626_1620_Stats.png
Attachment 13: 220627_1620_Temp.png
220627_1620_Temp.png
Attachment 14: 220626_1621_Bias.png
220626_1621_Bias.png
Attachment 15: R5_591_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260421862 (  840783.5 Hz)
        Other data format:    1498138 (    4836.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1449 (       4.7 Hz)
                               RESUME:       1448 (       4.7 Hz)
                              SYNC100:      33154 (     107.0 Hz)
                              WR48-63:      33154 (     107.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1428933 (    4613.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     452287 (    1460.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      309.737 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               13.828                0.000
  1               52.434                0.000
  2                0.366                0.000
  3                9.346                0.000
  4                0.001               63.918
  5                0.097                0.000
  6                0.290                0.000
  7                0.023                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31248979      29393          0          0        283        282       4085       4085          0      20658          0     112900
  1   66285610      18734          0          0        794        794       8573       8573          0          0          0      62107
  2   22411069     363765          0          0         53         53       2890       2890          0     357879          0      73563
  3   44202913     357046          0          0        221        221       5556       5556          0     345492          0      37941
  4    5178577       1284          0          0          1          1        641        641          0          0          0      37680
  5   19493707       4858          0          0         26         26       2403       2403          0          0          0      44391
  6   39593301     355717          0          0         57         57       5081       5081          0     345441          0      63694
  7   32007706     367341          0          0         14         14       3925       3925          0     359463          0      20011
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.789s (  765.751 blocks/s,  47.859 Mb/s)
Attachment 16: tmp_aida_stats.png
tmp_aida_stats.png
Attachment 17: tmp_aida_temps.png
tmp_aida_temps.png
Attachment 18: tmp_aida_hv.png
tmp_aida_hv.png
Attachment 19: R5_620
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261173934 (  672570.7 Hz)
        Other data format:     746069 (    1921.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        773 (       2.0 Hz)
                               RESUME:        773 (       2.0 Hz)
                              SYNC100:      33446 (      86.1 Hz)
                              WR48-63:      33446 (      86.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     677631 (    1745.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     225736 (     581.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      388.322 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                1.039                0.000
  1               21.090                0.000
  2                0.051                0.000
  3                0.148                0.000
  4                0.000               87.597
  5                0.001                0.000
  6                0.083                0.000
  7                0.003                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   25216948      33824          0          0         63         63       3334       3334          0      27030          0      55557
  1   75946108      20872          0          0        626        626       9810       9810          0          0          0      40146
  2   18060200     168976          0          0         23         23       2304       2304          0     164322          0      34386
  3   41452092     174351          0          0         26         26       5287       5287          0     163725          0      18377
  4    3680704        948          0          0          0          0        474        474          0          0          0      17387
  5   16518678       4150          0          0          1          1       2074       2074          0          0          0      20823
  6   44456577     169523          0          0         30         30       5686       5686          0     158091          0      29842
  7   35842627     173425          0          0          4          4       4477       4477          0     164463          0       9218
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.547s (  770.214 blocks/s,  48.138 Mb/s)
Attachment 20: 220624_2021_Stats.png
220624_2021_Stats.png
Attachment 21: 220626_2021_Temp.png
220626_2021_Temp.png
Attachment 22: 220626_2022_Bias.png
220626_2022_Bias.png
Attachment 23: R5_637_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260438015 (  849636.0 Hz)
        Other data format:    1481985 (    4834.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1460 (       4.8 Hz)
                               RESUME:       1460 (       4.8 Hz)
                              SYNC100:      33142 (     108.1 Hz)
                              WR48-63:      33142 (     108.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1412781 (    4609.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     447675 (    1460.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      306.529 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               15.951                0.000
  1               53.516                0.000
  2                0.642                0.000
  3                7.248                0.000
  4                0.000               53.241
  5                0.100                0.000
  6                0.344                0.000
  7                0.040                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31824689      29138          0          0        291        291       4129       4129          0      20298          0     110511
  1   66116092      18806          0          0        788        788       8615       8615          0          0          0      62706
  2   22609638     359670          0          0         68         68       2899       2899          0     353736          0      72536
  3   44231727     352857          0          0        199        199       5696       5696          0     341067          0      37873
  4    5159378       1270          0          0          0          0        635        635          0          0          0      37425
  5   19491876       4836          0          0         27         27       2391       2391          0          0          0      43832
  6   39561529     352144          0          0         68         68       4899       4899          0     342210          0      62927
  7   31443086     363264          0          0         19         19       3878       3878          0     355470          0      19865
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   42.891s (  746.084 blocks/s,  46.630 Mb/s)
Attachment 24: 220626_2211_Stats.png
220626_2211_Stats.png
Attachment 25: 220626_2211_Temps.png
220626_2211_Temps.png
Attachment 26: 220626_2212_Bias.png
220626_2212_Bias.png
Attachment 27: R5_658_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260445182 (  832450.2 Hz)
        Other data format:    1474818 (    4713.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1484 (       4.7 Hz)
                               RESUME:       1484 (       4.7 Hz)
                              SYNC100:      33166 (     106.0 Hz)
                              WR48-63:      33166 (     106.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1405518 (    4492.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     448659 (    1434.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.866 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               15.080                0.000
  1               57.798                0.000
  2                0.343                0.000
  3                7.278                0.000
  4                0.000               44.445
  5                0.093                0.000
  6                0.265                0.000
  7                0.017                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   32528426      29869          0          0        310        310       4234       4234          0      20781          0     110841
  1   66604779      18608          0          0        825        825       8479       8479          0          0          0      61991
  2   23919816     357449          0          0         63         63       3136       3136          0     351051          0      72944
  3   47288432     353667          0          0        196        196       5939       5939          0     341397          0      38095
  4    5032393       1218          0          0          0          0        609        609          0          0          0      37603
  5   19089927       4824          0          0         28         28       2384       2384          0          0          0      44213
  6   39392581     350027          0          0         52         52       5076       5076          0     339771          0      63206
  7   26588828     359156          0          0         10         10       3309       3309          0     352518          0      19766
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   42.102s (  760.067 blocks/s,  47.504 Mb/s)
  501   Sat Jun 25 22:48:57 2022 TDSunday 26 June 00:00-08:00
23.43 Check ASIC control
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

All system wide checks OK *except* WR & FPGA errors - attachments 1 & 2

DSSSD bias & leakage currents OK - attachment 3

adc data item stats - attachment 4

FEE64 temps OK - attachment 5

DSSSD bias & leakage currents OK - attachment 6

analysis of file S505/R5_415 - attachment 7
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 1%



02:36

analysis of file S505/R5_449 - attachment 8
 zero timewarps
 max deadtime aida02 c. 15%, all other FEE64s < 1%

per FEE64 rate & stat spectra - attachments 9 & 10

per p+n FEE64 1.8.L spectra - attachment 11
 aida01 pulser peak width 99 ch FWHM

per FEE64 1.8.H spectra - attachments 12 & 13

All system wide checks OK *except* WR & FPGA errors - attachments 14 & 15

adc data item stats - attachment 16

FEE64 temps OK - attachment 17

DSSSD bias & leakage currents OK - attachment 18


05:38

analysis of file S505/R5_483 - attachment 19
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 20 & 21

adc data item stats - attachment 22

FEE64 temps OK - attachment 23

DSSSD bias & leakage currents OK - attachment 24



07:27

analysis of file S505/R5_502 - attachment 25
 zero timewarps
 max deadtime aida02 c. 16%, all other FEE64s < 2%

All system wide checks OK *except* WR & FPGA errors - attachments 26 & 27

adc data item stats - attachment 28

FEE64 temps OK - attachment 29

DSSSD bias & leakage currents OK - attachment 30
Attachment 1: Screenshot_from_2022-06-25_23-48-13.png
Screenshot_from_2022-06-25_23-48-13.png
Attachment 2: Screenshot_from_2022-06-25_23-47-49.png
Screenshot_from_2022-06-25_23-47-49.png
Attachment 3: Screenshot_from_2022-06-25_23-46-30.png
Screenshot_from_2022-06-25_23-46-30.png
Attachment 4: Screenshot_from_2022-06-25_23-45-38.png
Screenshot_from_2022-06-25_23-45-38.png
Attachment 5: Screenshot_from_2022-06-25_23-43-15.png
Screenshot_from_2022-06-25_23-43-15.png
Attachment 6: Screenshot_from_2022-06-25_23-42-41.png
Screenshot_from_2022-06-25_23-42-41.png
Attachment 7: R5_415
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260603334 (  839394.5 Hz)
        Other data format:    1316666 (    4240.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1333 (       4.3 Hz)
                               RESUME:       1333 (       4.3 Hz)
                              SYNC100:      33156 (     106.8 Hz)
                              WR48-63:      33156 (     106.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1247688 (    4018.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     404733 (    1303.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      310.466 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                5.804                0.000
  1               49.047                0.000
  2                0.579                0.000
  3                4.786                0.000
  4                0.000               41.193
  5                0.052                0.000
  6                0.398                0.000
  7                0.178                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   29744429      29368          0          0        191        191       3879       3879          0      21228          0     101774
  1   62881123      17488          0          0        744        744       8000       8000          0          0          0      60446
  2   27391657     317618          0          0         76         76       3507       3507          0     310452          0      64539
  3   52390232     315971          0          0        228        228       6571       6571          0     302373          0      33153
  4    4668386       1144          0          0          0          0        572        572          0          0          0      32450
  5   16955242       4388          0          0         19         19       2175       2175          0          0          0      39119
  6   36767424     310864          0          0         60         60       4661       4661          0     301422          0      55985
  7   29804841     319825          0          0         15         15       3791       3791          0     312213          0      17267
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   42.203s (  758.238 blocks/s,  47.390 Mb/s)
Attachment 8: R5_449
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       9631
          ADC data format:   78467361 (  817544.9 Hz)
        Other data format:     362374 (    3775.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        372 (       3.9 Hz)
                               RESUME:        372 (       3.9 Hz)
                              SYNC100:       9989 (     104.1 Hz)
                              WR48-63:       9989 (     104.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     341652 (    3559.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     112408 (    1171.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       95.979 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                1.159                0.000
  1               13.633                0.000
  2                0.066                0.000
  3                0.790                0.000
  4                0.000               20.217
  5                0.003                0.000
  6                0.066                0.000
  7                0.003                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8759445       9043          0          0         45         45       1175       1175          0       6603          0      28377
  1   19225571       5374          0          0        215        215       2472       2472          0          0          0      17713
  2    7917505      86814          0          0         15         15       1017       1017          0      84750          0      17600
  3   15877732      87394          0          0         75         75       1976       1976          0      83292          0       9310
  4    1340586        302          0          0          0          0        151        151          0          0          0       8860
  5    4870076       1158          0          0          2          2        577        577          0          0          0      10559
  6   11177308      84854          0          0         17         17       1400       1400          0      82020          0      15171
  7    9299138      87435          0          0          3          3       1221       1221          0      84987          0       4818
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   10.232s (  941.224 blocks/s,  58.826 Mb/s)
Attachment 9: Screenshot_from_2022-06-26_02-36-29.png
Screenshot_from_2022-06-26_02-36-29.png
Attachment 10: Screenshot_from_2022-06-26_02-34-58.png
Screenshot_from_2022-06-26_02-34-58.png
Attachment 11: Screenshot_from_2022-06-26_02-33-46.png
Screenshot_from_2022-06-26_02-33-46.png
Attachment 12: Screenshot_from_2022-06-26_02-32-48.png
Screenshot_from_2022-06-26_02-32-48.png
Attachment 13: Screenshot_from_2022-06-26_02-31-49.png
Screenshot_from_2022-06-26_02-31-49.png
Attachment 14: Screenshot_from_2022-06-26_02-30-49.png
Screenshot_from_2022-06-26_02-30-49.png
Attachment 15: Screenshot_from_2022-06-26_02-30-25.png
Screenshot_from_2022-06-26_02-30-25.png
Attachment 16: Screenshot_from_2022-06-26_02-29-31.png
Screenshot_from_2022-06-26_02-29-31.png
Attachment 17: Screenshot_from_2022-06-26_02-28-59.png
Screenshot_from_2022-06-26_02-28-59.png
Attachment 18: Screenshot_from_2022-06-26_02-28-31.png
Screenshot_from_2022-06-26_02-28-31.png
Attachment 19: R5_483
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260645305 (  834521.2 Hz)
        Other data format:    1274695 (    4081.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1211 (       3.9 Hz)
                               RESUME:       1211 (       3.9 Hz)
                              SYNC100:      33162 (     106.2 Hz)
                              WR48-63:      33162 (     106.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1205949 (    3861.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     393822 (    1260.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.329 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                5.184                0.000
  1               48.274                0.000
  2                0.431                0.000
  3                3.375                0.000
  4                0.000               25.625
  5                0.027                0.000
  6                0.216                0.000
  7                0.008                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30200144      29499          0          0        166        166       3899       3899          0      21369          0      99521
  1   63245692      17566          0          0        710        710       8073       8073          0          0          0      60306
  2   25915590     306468          0          0         52         52       3383       3383          0     299598          0      62507
  3   52691198     306273          0          0        212        212       6559       6559          0     292731          0      32107
  4    4611863       1212          0          0          0          0        606        606          0          0          0      31071
  5   16567070       4140          0          0          9          9       2061       2061          0          0          0      37441
  6   37080833     301098          0          0         53         53       4819       4819          0     291354          0      54190
  7   30332915     308439          0          0          9          9       3762       3762          0     300897          0      16679
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.023s (  913.674 blocks/s,  57.105 Mb/s)
Attachment 20: Screenshot_from_2022-06-26_05-40-48.png
Screenshot_from_2022-06-26_05-40-48.png
Attachment 21: Screenshot_from_2022-06-26_05-40-20.png
Screenshot_from_2022-06-26_05-40-20.png
Attachment 22: Screenshot_from_2022-06-26_05-39-34.png
Screenshot_from_2022-06-26_05-39-34.png
Attachment 23: Screenshot_from_2022-06-26_05-38-30.png
Screenshot_from_2022-06-26_05-38-30.png
Attachment 24: Screenshot_from_2022-06-26_05-37-55.png
Screenshot_from_2022-06-26_05-37-55.png
Attachment 25: R5_502
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260592349 (  788577.6 Hz)
        Other data format:    1327651 (    4017.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1258 (       3.8 Hz)
                               RESUME:       1257 (       3.8 Hz)
                              SYNC100:      33231 (     100.6 Hz)
                              WR48-63:      33231 (     100.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1258674 (    3808.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     409569 (    1239.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      330.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               10.444                0.000
  1               49.479                0.000
  2                0.188                0.000
  3                4.232                0.000
  4                0.000               57.929
  5                0.053                0.000
  6                0.278                0.000
  7                0.015                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31755884      31086          0          0        242        242       4168       4168          0      22266          0     103011
  1   67198908      18739          0          0        724        723       8646       8646          0          0          0      63328
  2   19438169     317456          0          0         45         45       2440       2440          0     312486          0      65069
  3   45255369     318470          0          0        158        158       5750       5750          0     306654          0      33811
  4    4887217       1230          0          0          0          0        615        615          0          0          0      32106
  5   18040993       4688          0          0         19         19       2325       2325          0          0          0      38770
  6   42367504     314874          0          0         58         58       5348       5348          0     304062          0      56174
  7   31648305     321108          0          0         12         12       3939       3939          0     313206          0      17300
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.316s (  960.488 blocks/s,  60.030 Mb/s)
Attachment 26: Screenshot_from_2022-06-26_07-29-14.png
Screenshot_from_2022-06-26_07-29-14.png
Attachment 27: Screenshot_from_2022-06-26_07-28-54.png
Screenshot_from_2022-06-26_07-28-54.png
Attachment 28: Screenshot_from_2022-06-26_07-28-22.png
Screenshot_from_2022-06-26_07-28-22.png
Attachment 29: Screenshot_from_2022-06-26_07-27-50.png
Screenshot_from_2022-06-26_07-27-50.png
Attachment 30: Screenshot_from_2022-06-26_07-27-21.png
Screenshot_from_2022-06-26_07-27-21.png
  500   Sat Jun 25 17:06:42 2022 BA, AASaturday 25 June 2022 16:00-00:00

Took over the night shift from Magda

 

18:00 : attachments 1-3

20:00 : attachments 4-6

22:00 : attachments 7-9

00:00 : attachments 10-12

 

Attachment 1: Stat_17.png
Stat_17.png
Attachment 2: leak_17.png
leak_17.png
Attachment 3: Temp17.png
Temp17.png
Attachment 4: Temp_20.png
Temp_20.png
Attachment 5: Stat_20.png
Stat_20.png
Attachment 6: lek_20.png
lek_20.png
Attachment 7: stat_22.png
stat_22.png
Attachment 8: temp_22.png
temp_22.png
Attachment 9: lak_22.png
lak_22.png
  499   Sat Jun 25 06:55:07 2022 MSSaturday 25 June 2022 8:00-16:00

Took over the night shift from Tom.

7:00 The beam is back.

8:00 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

14:00 : attachments 10-12

16:00 : attachments 13-15

 

 

Attachment 1: Screenshot_2022-06-25_at_08-01-16_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_08-01-16_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-06-25_at_08-01-37_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_08-01-37_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-06-25_08-02-58.png
Screenshot_from_2022-06-25_08-02-58.png
Attachment 4: Screenshot_2022-06-25_at_10-00-41_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_10-00-41_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2022-06-25_at_10-01-03_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_10-01-03_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: Screenshot_from_2022-06-25_10-01-59.png
Screenshot_from_2022-06-25_10-01-59.png
Attachment 7: Screenshot_2022-06-25_at_12-00-54_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_12-00-54_Statistics_aidas-gsi.png
Attachment 8: Screenshot_2022-06-25_at_12-01-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_12-01-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_from_2022-06-25_12-02-08.png
Screenshot_from_2022-06-25_12-02-08.png
Attachment 10: Screenshot_2022-06-25_at_14-07-36_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_14-07-36_Statistics_aidas-gsi.png
Attachment 11: Screenshot_2022-06-25_at_14-07-59_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_14-07-59_Temperature_and_status_scan_aidas-gsi.png
Attachment 12: Screenshot_from_2022-06-25_14-08-50.png
Screenshot_from_2022-06-25_14-08-50.png
Attachment 13: Screenshot_2022-06-25_at_15-56-41_Statistics_aidas-gsi.png
Screenshot_2022-06-25_at_15-56-41_Statistics_aidas-gsi.png
Attachment 14: Screenshot_2022-06-25_at_15-56-57_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-25_at_15-56-57_Temperature_and_status_scan_aidas-gsi.png
Attachment 15: Screenshot_from_2022-06-25_15-58-15.png
Screenshot_from_2022-06-25_15-58-15.png
  498   Fri Jun 24 22:36:45 2022 TDSaturday 25 June 00:00-08:00
23:36 Check ASIC contorol
      Zero stats & all histograms

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x64 -> 0xa

all waveform AD9252 ADCs disabled
all fast discs disabled

BNC PB-5 settings (to p+n FEE64s only)
 amplitude 1.0V
 attenuator x1
 decay time 1ms
 polarity +
 frequency 22Hz

analysis of file S505/R5_154 - attachment 1
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

stats - attachments 2-6

per FEE64 1.8.H spectra - attachments 7 & 8

per p+n FEE64 1.8.L spectra - attachment 9
 aida01 pulser peak width 101 ch FWHM

per FEE64 stat & rate spectra - attachments 10 & 11

adc data item stats - attachment 12

FEE64 temps OK - attachment 13

DSSSD bias & leakage currents OK - attachments 14 & 15

00:20 all system wide checks OK *except* WR and FPGA errors - attachments 16 & 17


03:03

analysis of file S505/R5_189 - attachment 18
 zero timewarps
 max deadtime aida02 c. 17%, all other FEE64s < 5%

all system wide checks OK *except* WR and FPGA errors - attachments 19 & 20

adc data item stats - attachment 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage currents OK - attachment 23

03:30

per FEE64 1.8.H spectra - attcahments 24 & 25


06:09 no beam 

analysis of file S505/R5_221 - attachment 26
 zero timewarps
 all dead times << 1%

all system wide checks OK *except* WR and FPGA errors - attachments 27 & 28

adc data item stats - attachment 29

FEE64 temps OK - attachment 30

DSSSD bias & leakage currents OK - attachment 31
Attachment 1: R5_154
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260632197 (  880101.4 Hz)
        Other data format:    1287803 (    4348.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1443 (       4.9 Hz)
                               RESUME:       1443 (       4.9 Hz)
                              SYNC100:      33103 (     111.8 Hz)
                              WR48-63:      33103 (     111.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1218711 (    4115.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     391650 (    1322.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      296.139 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               14.978                0.000
  1               49.946                0.000
  2                0.925                0.000
  3                8.130                0.000
  4                0.131               17.535
  5                0.197                0.000
  6                0.760                0.000
  7                0.599                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   32916598      28529          0          0        287        287       4193       4193          0      19569          0      96572
  1   59287487      16678          0          0        710        710       7629       7629          0          0          0      57145
  2   20165095     309168          0          0         52         52       2618       2618          0     303828          0      63600
  3   42178844     305489          0          0        214        214       5322       5322          0     294417          0      32426
  4    5271956       1388          0          0          2          2        692        692          0          0          0      31943
  5   18164680       4584          0          0         28         28       2264       2264          0          0          0      38343
  6   41060613     306981          0          0         61         61       5168       5168          0     296523          0      54824
  7   41586924     314986          0          0         89         89       5217       5217          0     304374          0      16797
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   44.250s (  723.164 blocks/s,  45.198 Mb/s)
Attachment 2: Screenshot_from_2022-06-24_23-58-03.png
Screenshot_from_2022-06-24_23-58-03.png
Attachment 3: Screenshot_from_2022-06-24_23-56-54.png
Screenshot_from_2022-06-24_23-56-54.png
Attachment 4: Screenshot_from_2022-06-24_23-56-18.png
Screenshot_from_2022-06-24_23-56-18.png
Attachment 5: Screenshot_from_2022-06-24_23-55-53.png
Screenshot_from_2022-06-24_23-55-53.png
Attachment 6: Screenshot_from_2022-06-24_23-55-15.png
Screenshot_from_2022-06-24_23-55-15.png
Attachment 7: Screenshot_from_2022-06-24_23-54-37.png
Screenshot_from_2022-06-24_23-54-37.png
Attachment 8: Screenshot_from_2022-06-24_23-53-22.png
Screenshot_from_2022-06-24_23-53-22.png
Attachment 9: Screenshot_from_2022-06-24_23-51-35.png
Screenshot_from_2022-06-24_23-51-35.png
Attachment 10: Screenshot_from_2022-06-24_23-50-18.png
Screenshot_from_2022-06-24_23-50-18.png
Attachment 11: Screenshot_from_2022-06-24_23-49-08.png
Screenshot_from_2022-06-24_23-49-08.png
Attachment 12: Screenshot_from_2022-06-24_23-48-14.png
Screenshot_from_2022-06-24_23-48-14.png
Attachment 13: Screenshot_from_2022-06-24_23-47-40.png
Screenshot_from_2022-06-24_23-47-40.png
Attachment 14: Screenshot_from_2022-06-24_23-46-10.png
Screenshot_from_2022-06-24_23-46-10.png
Attachment 15: Screenshot_from_2022-06-24_23-44-08.png
Screenshot_from_2022-06-24_23-44-08.png
Attachment 16: Screenshot_from_2022-06-25_00-17-58.png
Screenshot_from_2022-06-25_00-17-58.png
Attachment 17: Screenshot_from_2022-06-25_00-17-36.png
Screenshot_from_2022-06-25_00-17-36.png
Attachment 18: R5_189
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260527466 (  871545.5 Hz)
        Other data format:    1392534 (    4658.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1393 (       4.7 Hz)
                               RESUME:       1393 (       4.7 Hz)
                              SYNC100:      33113 (     110.8 Hz)
                              WR48-63:      33113 (     110.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1323522 (    4427.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     429409 (    1436.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      298.926 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               14.659                0.000
  1               49.890                0.000
  2                0.405                2.804
  3                7.856                0.000
  4                0.000               45.637
  5                0.052                0.000
  6                0.331                0.000
  7                0.206                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   33437852      29241          0          0        298        298       4430       4430          0      19785          0     105011
  1   60873066      16872          0          0        713        713       7723       7723          0          0          0      63589
  2   19991278     336066          0          0         51         51       2502       2502          0     330960          0      70003
  3   42622163     327782          0          0        194        194       5471       5471          0     316452          0      35634
  4    5450705       1394          0          0          0          0        697        697          0          0          0      34276
  5   18430449       4608          0          0         17         17       2287       2287          0          0          0      41850
  6   41735438     335959          0          0         67         67       5311       5311          0     325203          0      60533
  7   37986515     340612          0          0         53         53       4692       4692          0     331122          0      18513
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.010s (  914.031 blocks/s,  57.127 Mb/s)
Attachment 19: Screenshot_from_2022-06-25_03-05-56.png
Screenshot_from_2022-06-25_03-05-56.png
Attachment 20: Screenshot_from_2022-06-25_03-05-35.png
Screenshot_from_2022-06-25_03-05-35.png
Attachment 21: Screenshot_from_2022-06-25_03-04-59.png
Screenshot_from_2022-06-25_03-04-59.png
Attachment 22: Screenshot_from_2022-06-25_03-04-24.png
Screenshot_from_2022-06-25_03-04-24.png
Attachment 23: Screenshot_from_2022-06-25_03-03-54.png
Screenshot_from_2022-06-25_03-03-54.png
Attachment 24: Screenshot_from_2022-06-25_03-31-28.png
Screenshot_from_2022-06-25_03-31-28.png
Attachment 25: Screenshot_from_2022-06-25_03-30-31.png
Screenshot_from_2022-06-25_03-30-31.png
Attachment 26: R5_221
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261819129 (  557711.5 Hz)
        Other data format:     100871 (     214.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        371 (       0.8 Hz)
                               RESUME:        371 (       0.8 Hz)
                              SYNC100:      33749 (      71.9 Hz)
                              WR48-63:      33749 (      71.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:      32631 (      69.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          2 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      469.453 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.835                0.000
  2                0.000               15.780
  3                0.000                0.000
  4                0.000              132.034
  5                0.000                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   18255571      37491          0          0          0          0       2430       2430          0      32631          0          1
  1   67878845      18770          0          0        371        371       9014       9014          0          0          0          0
  2    8103831       2048          0          0          0          0       1024       1024          0          0          0          1
  3   41453255      10648          0          0          0          0       5324       5324          0          0          0          0
  4    3384177        880          0          0          0          0        440        440          0          0          0          0
  5   15593071       3880          0          0          0          0       1940       1940          0          0          0          0
  6   55917021      14018          0          0          0          0       7009       7009          0          0          0          0
  7   51233358      13136          0          0          0          0       6568       6568          0          0          0          0
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.299s (  932.977 blocks/s,  58.311 Mb/s)
Attachment 27: Screenshot_from_2022-06-25_06-11-13.png
Screenshot_from_2022-06-25_06-11-13.png
Attachment 28: Screenshot_from_2022-06-25_06-10-50.png
Screenshot_from_2022-06-25_06-10-50.png
Attachment 29: Screenshot_from_2022-06-25_06-10-11.png
Screenshot_from_2022-06-25_06-10-11.png
Attachment 30: Screenshot_from_2022-06-25_06-09-40.png
Screenshot_from_2022-06-25_06-09-40.png
Attachment 31: Screenshot_from_2022-06-25_06-09-06.png
Screenshot_from_2022-06-25_06-09-06.png
  497   Fri Jun 24 15:04:06 2022 MarcFriday 24th June - evening shift

16:05 - Last checked was at 15:30. (see previous entry. All running smoothly.  

Next wide check will be in about an hour.

17:00

Stats -ok - attachment 1

Temperatures ok - attachement 2

Leakage current ok but - attachment 3

ucesb screen-shot - attachment 4

Wide check completed. Nothing different.

WR status decoder status: 
         Base         Current     Difference
aida07 fault      0xc53d :      0xc547 :      10  
aida08 fault      0xf1be :      0xf1e9 :      43  
White Rabbit error counter test result: Passed 6, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA timestamp::  
             Base         Current         Difference
aida07 fault      0x2a :      0x2c :      2  
FPGA Timestamp error counter test result: Passed 7, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Note that the leakage current is ok but has increased since yesterday , See graphana - attachement 5 - Prossibly due to higher beam intensity.

 

 

19:20 -

Stats  ok - attachment 6

Temp ok - attachment 7

Leakage current  ok - attachement 8

Wide check completed and same output as above.

 

22:00

Stats  ok - attachment 9

Temp ok - attachment 10

Leakage current  ok - attachement 11

Wide check completed and same output as above.

 

 

Attachment 1: Stats-Screenshot_from_2022-06-24_16-53-47.png
Stats-Screenshot_from_2022-06-24_16-53-47.png
Attachment 2: Temp-Screenshot_from_2022-06-24_16-52-58.png
Temp-Screenshot_from_2022-06-24_16-52-58.png
Attachment 3: HV_LC-Screenshot_from_2022-06-24_16-51-39.png
HV_LC-Screenshot_from_2022-06-24_16-51-39.png
Attachment 4: ucesb-Screenshot_from_2022-06-24_16-57-32.png
ucesb-Screenshot_from_2022-06-24_16-57-32.png
Attachment 5: GraphanaAIDA-Screenshot_from_2022-06-24_17-48-41.png
GraphanaAIDA-Screenshot_from_2022-06-24_17-48-41.png
Attachment 6: Stats-Screenshot_from_2022-06-24_19-21-32.png
Stats-Screenshot_from_2022-06-24_19-21-32.png
Attachment 7: Temp-Screenshot_from_2022-06-24_19-20-48.png
Temp-Screenshot_from_2022-06-24_19-20-48.png
Attachment 8: HV-LC-Screenshot_from_2022-06-24_19-19-43.png
HV-LC-Screenshot_from_2022-06-24_19-19-43.png
Attachment 9: Stats-Screenshot_from_2022-06-24_22-01-09.png
Stats-Screenshot_from_2022-06-24_22-01-09.png
Attachment 10: Temp-Screenshot_from_2022-06-24_22-01-50.png
Temp-Screenshot_from_2022-06-24_22-01-50.png
Attachment 11: HV_LC-Screenshot_from_2022-06-24_21-58-33.png
HV_LC-Screenshot_from_2022-06-24_21-58-33.png
  496   Fri Jun 24 06:56:06 2022 OHFriday 24 June 008:00-16:00
07:56 Took over from AM. No issues reported overnight
      Stats ok - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3
      
      System wide checks:
      Clocks ok
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc53b : 	 61  
      aida08 fault 	 0xf0e9 : 	 0xf1b7 : 	 206  
      White Rabbit error counter test result: Passed 6, Failed 2
      FPGA Errors
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x29 : 	 24  
      aida08 fault 	 0x1a : 	 0x2c : 	 18  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Analysis of R3_359 - attachment 4
      Dead time still around 15.9%

      Merger item rate around 2E6-4E6
      Tape server rate at 7750 kB/s
      Current HDD free space 1.7 TB
      Time left in HDD 2.59 days
      Will run out of space at some point on Sunday
      Have started compression of uncompressed raw data on the HDD. Using nice +10


08:45 AIDA07 dropped out from the merger at some point in the preceeding 5-10 minutes. - attachment 5
      Am regularly checking the statistics
      Was able to stop the DAQ by relaunching the merger.
      Reset the FEEs and recovered without a powercylce
      Started R4 following this stop

08:58 They have taken the beam to change the ion source for maybe 2 hours
      Will stop writing data but continue forwarding to MBS
      No storage ticked. Following break will be on R5


11:10 Beam is starting to come back R5 started
      Stats ok - attachment 6
      Temps ok - attachment 7
      Bias and leakage currents ok - attachment 8

13:05 Statistics ok - attachment 9
      Temps ok - attachment 10
      Bias and leakage curents ok - attachment 11
      System wide checks:
      WR
      		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc540 : 	 3  
      aida08 fault 	 0xf1be : 	 0xf1d0 : 	 18  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_33 - attachment 12

15:28 Statistics ok - attachment 13
      Temperature ok - attachment 14
      Bias and leakage current ok - attachment 15
      System wide checks:
      	
		 Base 		Current 	Difference
      aida07 fault 	 0xc53d : 	 0xc546 : 	 9  
      aida08 fault 	 0xf1be : 	 0xf1e0 : 	 34  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x2a : 	 0x2b : 	 1  
      FPGA Timestamp error counter test result: Passed 7, Failed 1

      Analysis of R5_63 - attachment 16
Attachment 1: 220624_0753_stats.png
220624_0753_stats.png
Attachment 2: 220624_0754_Temp.png
220624_0754_Temp.png
Attachment 3: 220624_0755_Bias.png
220624_0755_Bias.png
Attachment 4: R3_359_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260715255 (  897318.9 Hz)
        Other data format:    1204745 (    4146.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1323 (       4.6 Hz)
                               RESUME:       1323 (       4.6 Hz)
                              SYNC100:      33082 (     113.9 Hz)
                              WR48-63:      33082 (     113.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1135935 (    3909.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     353220 (    1215.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      290.549 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               10.171                0.000
  1               46.409                0.000
  2                1.006                0.000
  3                4.601                0.000
  4                0.000               37.574
  5                0.031                0.000
  6                0.191                0.000
  7                0.254                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903987      28078          0          0        236        236       4023       4023          0      19560          0      86851
  1   61208192      17252          0          0        718        718       7908       7908          0          0          0      52897
  2   26785987     291397          0          0         72         72       3425       3425          0     284403          0      56573
  3   43119726     289440          0          0        153        153       5319       5319          0     278496          0      29859
  4    4604412       1194          0          0          0          0        597        597          0          0          0      28791
  5   18336996       4656          0          0         16         16       2312       2312          0          0          0      34152
  6   36059875     277258          0          0         42         42       4514       4514          0     268146          0      48935
  7   39696080     295470          0          0         86         86       4984       4984          0     285330          0      15162
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.877s (  917.511 blocks/s,  57.344 Mb/s)
Attachment 5: 220624_0842_Stats_Stop.png
220624_0842_Stats_Stop.png
Attachment 6: 220624_1111_Stats.png
220624_1111_Stats.png
Attachment 7: 220624_1111_Temp.png
220624_1111_Temp.png
Attachment 8: 220624_1112_Bias.png
220624_1112_Bias.png
Attachment 9: 220624_1303_stats.png
220624_1303_stats.png
Attachment 10: 220624_1303_Temp.png
220624_1303_Temp.png
Attachment 11: 220624_1304_Bias.png
220624_1304_Bias.png
Attachment 12: R5_33_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260717099 (  869523.1 Hz)
        Other data format:    1202901 (    4011.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1344 (       4.5 Hz)
                               RESUME:       1344 (       4.5 Hz)
                              SYNC100:      33117 (     110.4 Hz)
                              WR48-63:      33117 (     110.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1133979 (    3782.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     354749 (    1183.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      299.839 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.504                0.000
  1               42.102                0.000
  2                0.275                0.000
  3                6.677                0.000
  4                0.000               31.976
  5                0.051                0.000
  6                0.250                0.000
  7                0.412                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   31758241      28867          0          0        238        238       4156       4156          0      20079          0      87480
  1   58453105      16104          0          0        635        635       7417       7417          0          0          0      53128
  2   20104679     289945          0          0         41         41       2565       2565          0     284733          0      56849
  3   41709424     287290          0          0        188        188       5262       5262          0     276390          0      29572
  4    5133297       1264          0          0          0          0        632        632          0          0          0      29278
  5   17857346       4470          0          0         22         22       2213       2213          0          0          0      34539
  6   40732502     278221          0          0         59         59       5175       5175          0     267753          0      48740
  7   44968505     296740          0          0        161        161       5697       5697          0     285024          0      15163
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.742s (  921.070 blocks/s,  57.567 Mb/s)
Attachment 13: 220624_1538_Stats.png
220624_1538_Stats.png
Attachment 14: 220624_1538_Temp.png
220624_1538_Temp.png
Attachment 15: 220624_1538_Bias.png
220624_1538_Bias.png
Attachment 16: R5_63_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260796374 (  834020.0 Hz)
        Other data format:    1123626 (    3593.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1191 (       3.8 Hz)
                               RESUME:       1191 (       3.8 Hz)
                              SYNC100:      33165 (     106.1 Hz)
                              WR48-63:      33165 (     106.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1054914 (    3373.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     350507 (    1120.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      312.698 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.889                0.000
  1               39.498                0.000
  2                0.306                0.000
  3                5.270                0.000
  4                0.000               26.707
  5                0.036                0.000
  6                0.217                0.000
  7                0.175                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30903831      29785          0          0        188        188       4074       4074          0      21261          0      87155
  1   61666403      16932          0          0        646        646       7820       7820          0          0          0      55280
  2   20140383     267008          0          0         42         42       2575       2575          0     261774          0      55290
  3   41652352     266495          0          0        164        164       5306       5306          0     255555          0      28879
  4    4871487       1204          0          0          0          0        602        602          0          0          0      28340
  5   17605547       4430          0          0         11         11       2204       2204          0          0          0      33538
  6   41056242     264471          0          0         55         55       5126       5126          0     254109          0      47278
  7   42900129     273301          0          0         85         85       5458       5458          0     262215          0      14747
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.305s (  774.730 blocks/s,  48.421 Mb/s)
  495   Fri Jun 24 00:34:55 2022 AMFriday 24 June 00:00-08:00

01:30    Attachments 1-5, white rabbit and fpga timestamp failures, otherwise all good

03:30    Attachments 11-15, white rabbit and fpga timestamp failures, otherwise all good

05:30    Attachments 6-10, white rabbit and fpga timestamp failures, otherwise all good

07:30    Attachments 16-20, white rabbit and fpga timestamp failures, otherwise all good

Attachment 1: 220624_0130_fpgatime.png
220624_0130_fpgatime.png
Attachment 2: 220624_0130_whiterabbit.png
220624_0130_whiterabbit.png
Attachment 3: 220624_0130_temps.png
220624_0130_temps.png
Attachment 4: 220624_0130_stats.png
220624_0130_stats.png
Attachment 5: 220624_0130_bias.png
220624_0130_bias.png
Attachment 6: 220624_0530_fpgatime.png
220624_0530_fpgatime.png
Attachment 7: 220624_0530_whiterabbit.png
220624_0530_whiterabbit.png
Attachment 8: 220624_0530_temps.png
220624_0530_temps.png
Attachment 9: 220624_0530_stats.png
220624_0530_stats.png
Attachment 10: 220624_0530_bias.png
220624_0530_bias.png
Attachment 11: 220624_0330_fpgatime.png
220624_0330_fpgatime.png
Attachment 12: 220624_0330_whiterabbit.png
220624_0330_whiterabbit.png
Attachment 13: 220624_0330_temps.png
220624_0330_temps.png
Attachment 14: 220624_0330_stats.png
220624_0330_stats.png
Attachment 15: 220624_0330_bias.png
220624_0330_bias.png
Attachment 16: 220624_0730_fpga.png
220624_0730_fpga.png
Attachment 17: 220624_0730_whiterabbit.png
220624_0730_whiterabbit.png
Attachment 18: 220624_0730_temps.png
220624_0730_temps.png
Attachment 19: 220624_0730_stats.png
220624_0730_stats.png
Attachment 20: 220624_0730_bias.png
220624_0730_bias.png
  494   Thu Jun 23 15:10:17 2022 Philippos PapadakisThursday 23 June 16:00-24:00
16:20
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc525 : 	 39  
aida08 fault 	 0xf0e9 : 	 0xf166 : 	 125  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
                 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x23 : 	 9  

Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)


18:05
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc527 : 	 41  
aida08 fault 	 0xf0e9 : 	 0xf177 : 	 142  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x24 : 	 10  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)


20:22
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf17f : 	 150  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x20 : 	 15  
aida08 fault 	 0x1a : 	 0x25 : 	 11  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)

21:56
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf186 : 	 157  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x26 : 	 12  

Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)

23:50
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc52e : 	 48  
aida08 fault 	 0xf0e9 : 	 0xf18c : 	 163  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
	
			 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x27 : 	 13  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15)
Attachment 1: Temperatures_2022-06-23_16-14-38.png
Temperatures_2022-06-23_16-14-38.png
Attachment 2: stats_2022-06-23_16-16-30.png
stats_2022-06-23_16-16-30.png
Attachment 3: IV2022-06-23_16-17-13.png
IV2022-06-23_16-17-13.png
Attachment 4: Temps_2022-06-23_18-08-04.png
Temps_2022-06-23_18-08-04.png
Attachment 5: Stats_2022-06-23_18-08-38.png
Stats_2022-06-23_18-08-38.png
Attachment 6: IV_2022-06-23_18-09-41.png
IV_2022-06-23_18-09-41.png
Attachment 7: Temps_2022-06-23_20-24-48.png
Temps_2022-06-23_20-24-48.png
Attachment 8: Stats_2022-06-23_20-25-18.png
Stats_2022-06-23_20-25-18.png
Attachment 9: IV2022-06-23_20-25-39.png
IV2022-06-23_20-25-39.png
Attachment 10: Temps_2022-06-23_21-59-26.png
Temps_2022-06-23_21-59-26.png
Attachment 11: Stats_2022-06-23_22-00-01.png
Stats_2022-06-23_22-00-01.png
Attachment 12: IV_2022-06-23_22-00-20.png
IV_2022-06-23_22-00-20.png
Attachment 13: Temps_2022-06-23_23-52-04.png
Temps_2022-06-23_23-52-04.png
Attachment 14: Stats_2022-06-23_23-52-32.png
Stats_2022-06-23_23-52-32.png
Attachment 15: IV_2022-06-23_23-52-50.png
IV_2022-06-23_23-52-50.png
  493   Thu Jun 23 06:56:02 2022 OHThursday 23 June 08:00-16:00
07:56 Taken over from AM
      Current White Rabbit Error status:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51a : 	 28  
      aida08 fault 	 0xf0e9 : 	 0xf132 : 	 73  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA error status:
      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  
      FPGA Timestamp error counter test result: Passed 6, Failed 2

      Likely that at some point AIDA08 will drop out again. I think this occurs if there is a sudden burst of WR errors which causes the merger to drop the link

      Statistics ok - attachment 1
      FEE temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3
      Analysis of file R3_93 - attachment 4
      Still deadtime in aida02 of ~15%
      Patrick has said he will look into whether we are still sending info code 4 and 5 for every ADC event which could be contributing to the deadtime.
      
      Currently 2.2 TB of space on /media/SecondDrive
      Current data rate of 6.23MB/s
      At current data rate this should last 4 days

08:27 Currently no beam
      Problem with Alvarez 3 now
      Tomorrow will have a 1 hour break due to changing of the source

09:36 Beam back

09:59 System wide checks
		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc51b : 	 29  
      aida08 fault 	 0xf0e9 : 	 0xf138 : 	 79  
      White Rabbit error counter test result: Passed 6, Failed 2

      			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x18 : 	 7  
      aida08 fault 	 0x1a : 	 0x20 : 	 6  

      Stats ok - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7

11:14 FRS group is changing from Hg setting back to Pt

11:36 Back on Pt R3_129

14:14 Stats ok - attachment 8
      Temperature ok - attachment 9
      Bias and leakage currents ok - attachment 10
      System wide checks:
      WR:
      		 Base 		Current 	Difference
      aida07 fault 	 0xc4fe : 	 0xc523 : 	 37  
      aida08 fault 	 0xf0e9 : 	 0xf156 : 	 109  
      White Rabbit error counter test result: Passed 6, Failed 2

      FPGA
      	
			 Base 		Current 		Difference
      aida07 fault 	 0x11 : 	 0x1b : 	 10  
      aida08 fault 	 0x1a : 	 0x20 : 	 6

      Analysis of R3_156 - attachment 11  
Attachment 1: 220623_0758_Stats.png
220623_0758_Stats.png
Attachment 2: 220623_0759_Temp.png
220623_0759_Temp.png
Attachment 3: 220623_0759_Bias.png
220623_0759_Bias.png
Attachment 4: R3_95_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260243782 (  817307.6 Hz)
        Other data format:    1676218 (    5264.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1230 (       3.9 Hz)
                               RESUME:       1231 (       3.9 Hz)
                              SYNC100:      33186 (     104.2 Hz)
                              WR48-63:      33186 (     104.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1607385 (    5048.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     440709 (    1384.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      318.416 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.258                0.000
  1               47.390                0.000
  2                1.040                0.000
  3                6.094                0.000
  4                0.003               59.113
  5                0.094                0.000
  6                0.304                0.000
  7                0.013                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   20778195      27751          0          0         46         46       2749       2749          0      22161          0      72140
  1   62611625      17407          0          0        705        706       7998       7998          0          0          0      66280
  2   31916579     410233          0          0         83         83       4122       4122          0     401823          0      94767
  3   52868439     408659          0          0        312        312       6874       6874          0     394287          0      28791
  4    5535248       1334          0          0          1          1        666        666          0          0          0      39988
  5   19704728       4912          0          0         25         25       2431       2431          0          0          0      52543
  6   35689179     392725          0          0         47         47       4446       4446          0     383739          0      67737
  7   31139789     413197          0          0         11         11       3900       3900          0     405375          0      18463
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.666s (  923.094 blocks/s,  57.693 Mb/s)
Attachment 5: 220623_0958_Stats.png
220623_0958_Stats.png
Attachment 6: 220623_0959_Temp.png
220623_0959_Temp.png
Attachment 7: 220623_1000_Bias.png
220623_1000_Bias.png
Attachment 8: 220623_1413_Stats.png
220623_1413_Stats.png
Attachment 9: 220623_1413_Temp.png
220623_1413_Temp.png
Attachment 10: 220623_1414_bias.png
220623_1414_bias.png
Attachment 11: R3_156_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260453729 (  852518.1 Hz)
        Other data format:    1466271 (    4799.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1362 (       4.5 Hz)
                               RESUME:       1362 (       4.5 Hz)
                              SYNC100:      33138 (     108.5 Hz)
                              WR48-63:      33138 (     108.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1397271 (    4573.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     435545 (    1425.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      305.511 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0               11.365                0.000
  1               54.806                0.000
  2                0.562                0.000
  3                5.946                0.000
  4                0.000               35.352
  5                0.068                0.000
  6                0.334                0.000
  7                0.054                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   30150644      28922          0          0        254        254       3938       3938          0      20538          0     108754
  1   64977250      18108          0          0        757        757       8297       8297          0          0          0      62895
  2   28023912     358551          0          0         83         83       3523       3523          0     351339          0      69361
  3   44222269     352853          0          0        180        180       5536       5536          0     341421          0      37017
  4    5005333       1272          0          0          0          0        636        636          0          0          0      36369
  5   19466878       5006          0          0         20         20       2483       2483          0          0          0      42187
  6   36422600     339861          0          0         37         37       4700       4700          0     330387          0      59710
  7   32184843     361698          0          0         31         31       4025       4025          0     353586          0      19252
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.418s (  903.496 blocks/s,  56.469 Mb/s)
  492   Wed Jun 22 23:18:45 2022 AM, OH, TD, MAThursday 23 June
00:00 TD Noticed on the last stats uploaded that AIDA08 had stopped sending signals.
      Attempted to recover restarting merger but this caused 01 to drop out.

00:10 DAQ recovered and AIDA01 recovered with a reboot


01:00 Attachments 1-4, white rabbit failures on aida07 and aida08, otherwise all good

03:00 Attachments 5-9, aida07 and aida08 failures on white rabbit and fpga timestamp, otherwise all good

05:00 Attachments 10-14, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good

07:00 Attachments 15-19, aida07 and aida08 failures on white rabbit and fpga timestamp continue, otherwise all good
Attachment 1: 220623_0100_whiterabbit.png
220623_0100_whiterabbit.png
Attachment 2: 220623_0100_temps.png
220623_0100_temps.png
Attachment 3: 220623_0100_stats.png
220623_0100_stats.png
Attachment 4: 220623_0100_bias.png
220623_0100_bias.png
Attachment 5: 220623_0300_whiterabbit.png
220623_0300_whiterabbit.png
Attachment 6: 220623_0300_fpgatime.png
220623_0300_fpgatime.png
Attachment 7: 220623_0300_temps.png
220623_0300_temps.png
Attachment 8: 220623_0300_stats.png
220623_0300_stats.png
Attachment 9: 220623_0300_bias.png
220623_0300_bias.png
Attachment 10: 220623_0500_fpgatime.png
220623_0500_fpgatime.png
Attachment 11: 220623_0500_whiterabbit.png
220623_0500_whiterabbit.png
Attachment 12: 220623_0500_temps.png
220623_0500_temps.png
Attachment 13: 220623_0500_stats.png
220623_0500_stats.png
Attachment 14: 220623_0500_bias.png
220623_0500_bias.png
Attachment 15: 220623_0700_fpgatime.png
220623_0700_fpgatime.png
Attachment 16: 220623_0700_whiterabbit.png
220623_0700_whiterabbit.png
Attachment 17: 220623_0700_temps.png
220623_0700_temps.png
Attachment 18: 220623_0700_stats.png
220623_0700_stats.png
Attachment 19: 220623_0700_bias.png
220623_0700_bias.png
  491   Wed Jun 22 15:09:37 2022 MAWednesday 22 June 16:00-00:00

16:00 Take over from Magda mrning shift

Beam was off of sometime for tunning to Mercurey setting.

17:00 attached 1-3

19:00 attached 4-6

21:00 attached 7-9

23:30 10-12

 

Attachment 1: BiasCurrent2022-06-22_17-11-07.png
BiasCurrent2022-06-22_17-11-07.png
Attachment 2: Statistics_2022-06-22_17-13-52.png
Statistics_2022-06-22_17-13-52.png
Attachment 3: Temperature2022-06-22_17-12-35.png
Temperature2022-06-22_17-12-35.png
Attachment 4: BiasCurrent2022-06-22_19-00-31.png
BiasCurrent2022-06-22_19-00-31.png
Attachment 5: Temperature2022-06-22_19-02-18.png
Temperature2022-06-22_19-02-18.png
Attachment 6: Statics2022-06-22_19-03-11.png
Statics2022-06-22_19-03-11.png
Attachment 7: BiasCurrent2022-06-22_21-04-38.png
BiasCurrent2022-06-22_21-04-38.png
Attachment 8: Statistics2022-06-22_21-07-07.png
Statistics2022-06-22_21-07-07.png
Attachment 9: Temperature2022-06-22_21-05-51.png
Temperature2022-06-22_21-05-51.png
Attachment 10: BiasCurrent2022-06-22_23-29-40.png
BiasCurrent2022-06-22_23-29-40.png
Attachment 11: Statistics2022-06-22_23-31-30.png
Statistics2022-06-22_23-31-30.png
Attachment 12: Temperature2022-06-22_23-30-28.png
Temperature2022-06-22_23-30-28.png
  490   Wed Jun 22 10:09:12 2022 PJCSINFO: FEE64 supply Voltages

Study of the FEE64 power supply distribution has yielded the following :-

The most sensitive regulator, as regards the device voltage input, is the TPS51100 that supplies the DDR memory reference voltages. This requires +4.75 to +5.25 volts. This is supplied from the Power input connector +5v input.

The common LT3080 regulator used over much of the board is a Low Voltage Dropout regulator. This requires 0.5v difference between input and output voltage as minimum. This is not a problem with the +4.75v minimum for the TPS51100 requirement setting the voltage for the board.

The supply to the mezzanine is direct from the power connector +5v input. On the mezzanine there is an LT3080 for each ASIC supplying the required 3.3v. These regulators would possibly benefit from a 1uF capacitor at the Control voltage input.

The simplest approach would be to add a capacitor to the bottom layer where the Control voltage enters the mezzanine. 

 

The power cable has a nominal resistance of 13.3ohms/km. The 3 conductors of the cable are supplying 10A when all is in operation. So the expected voltage drop would thus be ( 10 x 13.3 x 0.007 ) /3 => 0.3v each core. 

The conclusion would be that the voltage at the power supply should drop to 5.25 v thus ensuring the TPS51100 is supplied as required regardless of the operation of the FEE. 

This will be tested at Daresbury.

  489   Wed Jun 22 07:12:46 2022 MSWednesday 22 June 8:00-16:00

Took over the night shift from Tom.

8:10 : attachments 1-3

10:00 : attachments 4-6

12:00 : attachments 7-9

The beam was off for around 20' starting at around 11:30-11:50 due to ion source issues.

12:19 OH Disabled all discriminators to reduce the data rate.

13:06 Analysis of R2_104 15.66% deadtime on AIDA04- attachment 10

14:00 : attachments 11-13

16:00 : attachments 14-16

 

Attachment 1: Screenshot_from_2022-06-22_08-11-30.png
Screenshot_from_2022-06-22_08-11-30.png
Attachment 2: Screenshot_2022-06-22_at_08-10-58_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_08-10-58_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2022-06-22_at_08-10-16_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_08-10-16_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2022-06-22_at_09-59-11_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_09-59-11_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2022-06-22_at_09-59-40_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_09-59-40_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: Screenshot_from_2022-06-22_10-00-13.png
Screenshot_from_2022-06-22_10-00-13.png
Attachment 7: Screenshot_2022-06-22_at_11-58-57_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_11-58-57_Statistics_aidas-gsi.png
Attachment 8: Screenshot_2022-06-22_at_11-59-18_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_11-59-18_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_from_2022-06-22_11-59-55.png
Screenshot_from_2022-06-22_11-59-55.png
Attachment 10: R2_104_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260553751 (  867725.3 Hz)
        Other data format:    1366249 (    4550.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1464 (       4.9 Hz)
                               RESUME:       1464 (       4.9 Hz)
                              SYNC100:      33119 (     110.3 Hz)
                              WR48-63:      33119 (     110.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1297083 (    4319.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     518974 (    1728.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      300.272 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                7.776                0.000
  1               47.563                0.000
  2                0.466                0.000
  3                4.438                0.000
  4                0.149               14.780
  5               16.612                0.000
  6                0.972                0.000
  7                0.380                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   27837591      27876          0          0        187        187       3557       3557          0      20388          0     104437
  1   57538232      16015          0          0        651        652       7356       7356          0          0          0      64114
  2   23047755     331545          0          0         63         63       2949       2949          0     325521          0      64002
  3   44324924     329582          0          0        146        145       5671       5671          0     317949          0      37242
  4   11203550       2766          0          0         57         57       1326       1326          0          0          0      90620
  5   28839527       7636          0          0        219        219       3599       3599          0          0          0      65648
  6   37663072     317251          0          0         87         87       4772       4772          0     307533          0      63360
  7   30099100     333578          0          0         54         54       3889       3889          0     325692          0      29551
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.320s (  932.392 blocks/s,  58.275 Mb/s)
Attachment 11: Screenshot_2022-06-22_at_14-01-58_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_14-01-58_Statistics_aidas-gsi.png
Attachment 12: Screenshot_2022-06-22_at_14-02-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_14-02-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 13: Screenshot_from_2022-06-22_14-03-14.png
Screenshot_from_2022-06-22_14-03-14.png
Attachment 14: Screenshot_2022-06-22_at_15-57-21_Statistics_aidas-gsi.png
Screenshot_2022-06-22_at_15-57-21_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2022-06-22_at_15-57-42_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-06-22_at_15-57-42_Temperature_and_status_scan_aidas-gsi.png
Attachment 16: Screenshot_from_2022-06-22_15-58-16.png
Screenshot_from_2022-06-22_15-58-16.png
  488   Tue Jun 21 23:29:16 2022 TDWednesday 22 June 00:00-08:00
00.28 all histograms & stats zero'd
      baseline counters

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

BNC PB-5 pulser
 to p+n FEE64s only
  amplitude 1.0V
  attenuator x1
  decay time 1ms
  polarity +
  frequency 2Hz

all AD9252 waveform ADCs disabled

Tape Server -> no storage mode

00.36 check ASIC control all FEE64s all ASICs

      all system wide checks OK

00.41 BNC PB-5 pulser rate increased from 2Hz to 22Hz per request by HA/NH - to ensure DTAS correlation checker has sufficient rate of identified pulser events 

per p+n FEE64 1.8.L spectra - attachment 1
 aida01 pulser peak width 88 ch FWHM

per FEE64 rate spectra - attachment 2

adc, pause and correlation scaler data items - push, aida01 and aida02 stats - attachments 3-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage current OK - attachments 10 & 11

02.28 aida05 zero adc data items, unable to halt aida05 - all other FEE64s OK

      restart NewMerger -> now able to halt aida05

      DAQ go OK, merge rate c. 1.4M data items/s

per FEE64 1.8.H spectra - attachments 12 & 13
 approx flat distribution to c. 10GeV


adc data items stats OK - attachments 14

FEE64 temps OK - attachment 15

DSSSD bias & leakage current OK - attachments 16 & 17

02.55 Some problems with GO4 data analysis for PID (FRS PID looks OK). 204Pt setting. Will commence writing data to disk file S505/R2_0

04.12 per FEE64 1.8.H spectra - attachments 18 & 19
      c. 9GeV peak in upstream DSSSDs (aida01, 2, 3 & 4) indicating (high energy) heavy fragments are stopping
      Low energy events near c. 200MeV lower mass, high energy, low energy loss - fission fragments?

04.20 per FEE64 stats spectra - attachment 20
      indicates implant x-y distribution - off centre (more significantly in x than in y) but DSSSDs appear to see most of distribution
      remember raw distro will have significant fraction of fission fragments - may be misleading - need to gate on high energy events to see x-y distro of ions of 
interest

adc data items stats OK - attachments 21

FEE64 temps OK - attachment 22

DSSSD bias & leakage current OK - attachments 23



06.10 per FEE64 rate spectra - attachment 24

per FEE64 1.8.H spectra - attachments 25 & 26

per FEE64 stats spectra - attachment 27
     
adc data items stats OK - attachments 28

FEE64 temps OK - attachment 29

system wide checks OK *except* WR & FPGA errors - see attachments 30 & 31

DSSSD bias & leakage current OK - attachments 32

07.52 analysis of file S505/R2_47 - attachment 33
      all FEE64 dead times < 1% *except* aida02 c. 7%
Attachment 1: Screenshot_from_2022-06-22_00-50-20.png
Screenshot_from_2022-06-22_00-50-20.png
Attachment 2: Screenshot_from_2022-06-22_00-49-07.png
Screenshot_from_2022-06-22_00-49-07.png
Attachment 3: Screenshot_from_2022-06-22_00-47-56.png
Screenshot_from_2022-06-22_00-47-56.png
Attachment 4: Screenshot_from_2022-06-22_00-46-57.png
Screenshot_from_2022-06-22_00-46-57.png
Attachment 5: Screenshot_from_2022-06-22_00-46-17.png
Screenshot_from_2022-06-22_00-46-17.png
Attachment 6: Screenshot_from_2022-06-22_00-45-33.png
Screenshot_from_2022-06-22_00-45-33.png
Attachment 7: Screenshot_from_2022-06-22_00-44-58.png
Screenshot_from_2022-06-22_00-44-58.png
Attachment 8: Screenshot_from_2022-06-22_00-44-13.png
Screenshot_from_2022-06-22_00-44-13.png
Attachment 9: Screenshot_from_2022-06-22_00-36-06.png
Screenshot_from_2022-06-22_00-36-06.png
Attachment 10: Screenshot_from_2022-06-22_00-35-17.png
Screenshot_from_2022-06-22_00-35-17.png
Attachment 11: Screenshot_from_2022-06-22_00-34-01.png
Screenshot_from_2022-06-22_00-34-01.png
Attachment 12: Screenshot_from_2022-06-22_02-44-02.png
Screenshot_from_2022-06-22_02-44-02.png
Attachment 13: Screenshot_from_2022-06-22_02-41-13.png
Screenshot_from_2022-06-22_02-41-13.png
Attachment 14: Screenshot_from_2022-06-22_02-40-14.png
Screenshot_from_2022-06-22_02-40-14.png
Attachment 15: Screenshot_from_2022-06-22_02-39-42.png
Screenshot_from_2022-06-22_02-39-42.png
Attachment 16: Screenshot_from_2022-06-22_02-39-01.png
Screenshot_from_2022-06-22_02-39-01.png
Attachment 17: Screenshot_from_2022-06-22_02-38-24.png
Screenshot_from_2022-06-22_02-38-24.png
Attachment 18: Screenshot_from_2022-06-22_04-10-47.png
Screenshot_from_2022-06-22_04-10-47.png
Attachment 19: Screenshot_from_2022-06-22_04-10-00.png
Screenshot_from_2022-06-22_04-10-00.png
Attachment 20: Screenshot_from_2022-06-22_04-23-13.png
Screenshot_from_2022-06-22_04-23-13.png
Attachment 21: Screenshot_from_2022-06-22_04-20-06.png
Screenshot_from_2022-06-22_04-20-06.png
Attachment 22: Screenshot_from_2022-06-22_04-19-34.png
Screenshot_from_2022-06-22_04-19-34.png
Attachment 23: Screenshot_from_2022-06-22_04-19-00.png
Screenshot_from_2022-06-22_04-19-00.png
Attachment 24: Screenshot_from_2022-06-22_06-10-41.png
Screenshot_from_2022-06-22_06-10-41.png
Attachment 25: Screenshot_from_2022-06-22_06-09-33.png
Screenshot_from_2022-06-22_06-09-33.png
Attachment 26: Screenshot_from_2022-06-22_06-08-08.png
Screenshot_from_2022-06-22_06-08-08.png
Attachment 27: Screenshot_from_2022-06-22_06-06-50.png
Screenshot_from_2022-06-22_06-06-50.png
Attachment 28: Screenshot_from_2022-06-22_06-05-58.png
Screenshot_from_2022-06-22_06-05-58.png
Attachment 29: Screenshot_from_2022-06-22_06-05-20.png
Screenshot_from_2022-06-22_06-05-20.png
Attachment 30: Screenshot_from_2022-06-22_06-04-27.png
Screenshot_from_2022-06-22_06-04-27.png
Attachment 31: Screenshot_from_2022-06-22_06-03-53.png
Screenshot_from_2022-06-22_06-03-53.png
Attachment 32: Screenshot_from_2022-06-22_06-03-05.png
Screenshot_from_2022-06-22_06-03-05.png
Attachment 33: R2_47
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259428432 (  714533.4 Hz)
        Other data format:    2491568 (    6862.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        881 (       2.4 Hz)
                               RESUME:        880 (       2.4 Hz)
                              SYNC100:      33352 (      91.9 Hz)
                              WR48-63:      33352 (      91.9 Hz)
                           FEE64 disc:    1422279 (    3917.3 Hz)
                             MBS info:    1000824 (    2756.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     416179 (    1146.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      363.074 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.857                0.000
  1               26.452                0.000
  2                0.132                0.000
  3                1.570                0.000
  4                0.002               46.041
  5                0.661                0.000
  6                0.312                0.000
  7                0.027                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   25701314     249377          0          0         56         56       3411       3411     217219      25224          0      83012
  1   63351849     247459          0          0        567        566       8361       8361     229604          0          0      62244
  2   23166597     387869          0          0         37         37       3010       3010     134065     247710          0      48973
  3   48876539     363237          0          0        105        105       6171       6171     104940     245745          0      28921
  4    8006429     267186          0          0          3          3       1045       1045     265090          0          0      68788
  5   22095663     170195          0          0         36         36       2789       2789     164545          0          0      52757
  6   40483353     472437          0          0         66         66       5106       5106     227985     234108          0      48544
  7   27746688     333808          0          0         11         11       3459       3459      78831     248037          0      22940
  8          0          0          0          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   35.072s (  912.402 blocks/s,  57.025 Mb/s)
  487   Tue Jun 21 10:19:05 2022 OH MLTuesday 21st June 08:00-00:00
11:19 Stats ok 0xa on all FEE - attachment 1
      Current pulser settings - attachment 2
      FEE Temperatures ok - attachment 3
      Noticed both AIDA03 and AIDA07 had ASICs with missing channels. Did a check ASIC control on them to recover
      Layout 1 after recovering channels - attachment 4
      New stats (Note increased rate on 3 and 7) attachment 5
      Bias and leakage currents ok - attachment 6

      S4 now in closed access. Trying to find out if the beam dump has been put in front of AIDA. Will make sure this is checked before beam to S4.

14:05 Have had it confirmed that the flange beam dump is in position in S4

15:43 Stats ok -attachment 7
      Temp ok - attachment 8
      Bias and Leakage current ok - attachment 9


17:45 Currently the time machine trigger(for correlation) is connected to the DTAS or.
      As the beam is tuning they are triggering at 150k. This is 450k worth of scalers in AIDA
      During the experiment this will be changed to SC1 trigger to solve this.

19:49: Stats ok - attachment 10
       temperatures ok - attachment 11
       Bias - Leakage current ok - attachment 12
Attachment 1: 220621_1119_Stats.png
220621_1119_Stats.png
Attachment 2: 220621_1120_Pulser.png
220621_1120_Pulser.png
Attachment 3: 220621_1120_Temp.png
220621_1120_Temp.png
Attachment 4: 220621_1123_Layout1.png
220621_1123_Layout1.png
Attachment 5: 220621_1123_Stats.png
220621_1123_Stats.png
Attachment 6: 220621_1123_Bias.png
220621_1123_Bias.png
Attachment 7: 220621_1542_Stats.png
220621_1542_Stats.png
Attachment 8: 220621_1542_Temp.png
220621_1542_Temp.png
Attachment 9: 220621_1543_Bias.png
220621_1543_Bias.png
Attachment 10: Stats-Screenshot_from_2022-06-21_19-50-19.png
Stats-Screenshot_from_2022-06-21_19-50-19.png
Attachment 11: Temp-Screenshot_from_2022-06-21_19-48-56.png
Temp-Screenshot_from_2022-06-21_19-48-56.png
Attachment 12: HV_LC-Screenshot_from_2022-06-21_19-47-49.png
HV_LC-Screenshot_from_2022-06-21_19-47-49.png
  486   Mon Jun 20 09:26:23 2022 OH, TDMonday 20 June
10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11
Attachment 1: 220620_1116_stats_no_waveform.png
220620_1116_stats_no_waveform.png
Attachment 2: 220620_1223_stats_no_waveform.png
220620_1223_stats_no_waveform.png
Attachment 3: Screenshot_from_2022-06-20_21-17-41.png
Screenshot_from_2022-06-20_21-17-41.png
Attachment 4: Screenshot_from_2022-06-20_21-14-12.png
Screenshot_from_2022-06-20_21-14-12.png
Attachment 5: Screenshot_from_2022-06-20_21-12-06.png
Screenshot_from_2022-06-20_21-12-06.png
Attachment 6: Screenshot_from_2022-06-20_21-10-55.png
Screenshot_from_2022-06-20_21-10-55.png
Attachment 7: Screenshot_from_2022-06-20_21-08-58.png
Screenshot_from_2022-06-20_21-08-58.png
Attachment 8: Screenshot_from_2022-06-20_21-04-58.png
Screenshot_from_2022-06-20_21-04-58.png
Attachment 9: Screenshot_from_2022-06-20_21-04-00.png
Screenshot_from_2022-06-20_21-04-00.png
Attachment 10: Screenshot_from_2022-06-20_21-02-07.png
Screenshot_from_2022-06-20_21-02-07.png
Attachment 11: Screenshot_from_2022-06-20_21-01-07.png
Screenshot_from_2022-06-20_21-01-07.png
  485   Sun Jun 19 19:55:20 2022 NHaida03 and aida04
aida03 and aida04 seem to stop sending data  quickly

stopping DAQ + merger and restarting got them resending data, but quickly stopped again

both are showing large WR errors that increases approx every second.

Assume HDMI cables not well seated and causing all sorts of odd behaviour
Attachment 1: Screenshot_2022-06-19_205528.png
Screenshot_2022-06-19_205528.png
Attachment 2: Screenshot_2022-06-19_205544.png
Screenshot_2022-06-19_205544.png
  484   Sun Jun 19 12:36:10 2022 OHSunday 19th June
Yesterday the sum/inverter was found to be a considerable source of noise in the system.

Plugging it into an oscilloscope this morning it was clear there was a Vpp 17mv 60kHz sin wave in its signal - attachment 1 yellow trace

Moved the inverter to a crate in the messheute and did not see same behaviour. This time see Vpp 5mV and a ~265Hz wave that looks much less clean. - attachment 2 yellow

Put inverter back in CAEN crate in S4 and confirm similar behavior.

Move CAEN crate to new mains port, same behaviour

Move CAEN crate to AC voltage stabiliser, same behaviour

Remove module one by one from CAEN crate.
Removing BIAS supply - No change
Remove Pulser - No change
Remove right most MACB - No change
Remove second right most MACB - Signal disappears  - attachment 3 blue trace
Remove second right most MACB from the rack entirely
Put right most MACB back - Signal comes back - attachment 4 blue trace
Test MACB with and without HDMI connectors connected signal remains
Remove rightmost MACB from the rack - Signal disappears

Put pulser and bias back - Signal does not return
Signal also doesn't return when ramping up bias or running the pulser
Power DAQ up - No change in performance from yesterday with inverter removed.

MACB and Inverter don't share any voltages on NIM:
MACB +-6V and Gnd
Inverter +-24V and Gnd

CAEN Crate does not have any easily accessible monitoring points. Anyone have any ideas?

MACBs and inverter checked on another crate - no signal

Inverter put back in CAEN crate and no noise observed on scope.
With waveforms disabled am able to get <100kHz on all FEEs with the pulser connected

Tested adding bias filters - No change

15:30 Nic realised that one of the mains boxes is labelled measurement network so we connect the AIDA PSU to this via the ac stabiliser
      We get ok stats even with waveforms on at 0xa- attachment 5

      Waveforms still have the 100kHz oscillation in them though attachments 6 and 7

      Without waveforms we get similar performance to before the switch to the new mains - attachment 8

15:56 Move CAEN NIM crate across to this mains network
      300+kHz on all FEEs with waveforms on. Can recover waveform off performance though

16:01 Place CAEN NIM crate back onto platform mains but don't recover the ealier rates with waveform. Everything still around 300kHz.

17:43 Current best settings are waveforms off and we can run at 0xa on all FEEs - attachment 9
Attachment 1: 1.jpg
1.jpg
Attachment 2: 2.jpg
2.jpg
Attachment 3: 3.jpg
3.jpg
Attachment 4: 4.jpg
4.jpg
Attachment 5: 220618_1753_stats_no_waveform.png
220618_1753_stats_no_waveform.png
  483   Sun Jun 19 00:22:24 2022 TDSunday 19 June
01.20 

ASIC settings 2021Apr29-13-16-00
 *except* slow comparator 0x64 -> 0xa

All waveform ADCs disabled

BNC PB-5 pulser
 to p+n FEE64s only

all system wide checks OK *except* ADC calibration, WR decoder & FPGA timestamp errors - see attachments 1 & 2

DSSSD bias & leakage current OK - see attachments 3 & 4

FEE64 temps OK - see attachment 5

statistics OK - ADC data, pause, resume & correlation scaler data items
                data push, data flush, aida01, aida02 - see attachments 6-13

per FEE64 1.8.L spectra - see attachment 14
 aida01 pulser peak width 73 ch FWHM
Attachment 1: Screenshot_from_2022-06-19_01-24-50.png
Screenshot_from_2022-06-19_01-24-50.png
Attachment 2: Screenshot_from_2022-06-19_01-24-24.png
Screenshot_from_2022-06-19_01-24-24.png
Attachment 3: Screenshot_from_2022-06-19_01-20-50.png
Screenshot_from_2022-06-19_01-20-50.png
Attachment 4: Screenshot_from_2022-06-19_01-20-09.png
Screenshot_from_2022-06-19_01-20-09.png
Attachment 5: Screenshot_from_2022-06-19_01-19-37.png
Screenshot_from_2022-06-19_01-19-37.png
Attachment 6: Screenshot_from_2022-06-19_01-18-35.png
Screenshot_from_2022-06-19_01-18-35.png
Attachment 7: Screenshot_from_2022-06-19_01-18-05.png
Screenshot_from_2022-06-19_01-18-05.png
Attachment 8: Screenshot_from_2022-06-19_01-17-28.png
Screenshot_from_2022-06-19_01-17-28.png
Attachment 9: Screenshot_from_2022-06-19_01-15-44.png
Screenshot_from_2022-06-19_01-15-44.png
Attachment 10: Screenshot_from_2022-06-19_01-14-59.png
Screenshot_from_2022-06-19_01-14-59.png
Attachment 11: Screenshot_from_2022-06-19_01-14-24.png
Screenshot_from_2022-06-19_01-14-24.png
Attachment 12: Screenshot_from_2022-06-19_01-13-43.png
Screenshot_from_2022-06-19_01-13-43.png
Attachment 13: Screenshot_from_2022-06-19_01-13-03.png
Screenshot_from_2022-06-19_01-13-03.png
Attachment 14: Screenshot_from_2022-06-19_01-12-16.png
Screenshot_from_2022-06-19_01-12-16.png
  482   Sat Jun 18 09:46:57 2022 OH, NHFriday 17th June
Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23
Attachment 1: 220617_1322_stats.png
220617_1322_stats.png
Attachment 2: 220617_1420_stats.png
220617_1420_stats.png
Attachment 3: 220617_11428_stats.png
220617_11428_stats.png
Attachment 4: 220617_1428_layout1.png
220617_1428_layout1.png
Attachment 5: 220617_1440_stats_waveform_on.png
220617_1440_stats_waveform_on.png
Attachment 6: 220617_1444_stats_no_waveform.png
220617_1444_stats_no_waveform.png
Attachment 7: 220617_1459_stats_waveform_on.png
220617_1459_stats_waveform_on.png
Attachment 8: 220617_1517_stats_no_waveform.png
220617_1517_stats_no_waveform.png
Attachment 9: 220617_1624_stats_waveform_on.png
220617_1624_stats_waveform_on.png
Attachment 10: 220617_1626_stats_waveform_off.png
220617_1626_stats_waveform_off.png
Attachment 11: 220617_1649_stats_waveform_on.png
220617_1649_stats_waveform_on.png
Attachment 12: 220617_1652_no_waveform.png
220617_1652_no_waveform.png
Attachment 13: 220617_1709_waveform_on.png
220617_1709_waveform_on.png
Attachment 14: 220617_1724_stats_no_waveform.png
220617_1724_stats_no_waveform.png
Attachment 15: 220617_1729_stats_waveform_on.png
220617_1729_stats_waveform_on.png
Attachment 16: 220617_1733_stats_no_waveform.png
220617_1733_stats_no_waveform.png
Attachment 17: 220617_1736_0x21.png
220617_1736_0x21.png
Attachment 18: 220617_1738_0x14.png
220617_1738_0x14.png
Attachment 19: 220617_1738_0x12.png
220617_1738_0x12.png
Attachment 20: 220617_1807_stats_waveform_on.png
220617_1807_stats_waveform_on.png
Attachment 21: 220617_1832_stats_no_waveform.png
220617_1832_stats_no_waveform.png
Attachment 22: 220617_1844_0xf.png
220617_1844_0xf.png
Attachment 23: 220617_1845_stats_waveform_on.png
220617_1845_stats_waveform_on.png
  481   Tue Jun 14 12:15:33 2022 NHS450 Archived
All MIDAS files for S450 (/TapeData/S450) have been copied to lustre and archived 
Attachment 1: s450_aida_tape.txt
R1_0.gz: (0x00000009) exists archived, archive_id:13
R1_100.gz: (0x00000009) exists archived, archive_id:13
R1_101.gz: (0x00000009) exists archived, archive_id:13
R1_102.gz: (0x00000009) exists archived, archive_id:13
R1_103.gz: (0x00000009) exists archived, archive_id:13
R1_104.gz: (0x00000009) exists archived, archive_id:13
R1_105.gz: (0x00000009) exists archived, archive_id:13
R1_106.gz: (0x00000009) exists archived, archive_id:13
R1_107.gz: (0x00000009) exists archived, archive_id:13
R1_108.gz: (0x00000009) exists archived, archive_id:13
R1_109.gz: (0x00000009) exists archived, archive_id:13
R1_10.gz: (0x00000009) exists archived, archive_id:13
R1_110.gz: (0x00000009) exists archived, archive_id:13
R1_111.gz: (0x00000009) exists archived, archive_id:13
R1_112.gz: (0x00000009) exists archived, archive_id:13
R1_113.gz: (0x00000009) exists archived, archive_id:13
R1_114.gz: (0x00000009) exists archived, archive_id:13
R1_115.gz: (0x00000009) exists archived, archive_id:13
R1_116.gz: (0x00000009) exists archived, archive_id:13
R1_117.gz: (0x00000009) exists archived, archive_id:13
R1_118.gz: (0x00000009) exists archived, archive_id:13
R1_119.gz: (0x00000009) exists archived, archive_id:13
R1_11.gz: (0x00000009) exists archived, archive_id:13
R1_120.gz: (0x00000009) exists archived, archive_id:13
R1_121.gz: (0x00000009) exists archived, archive_id:13
R1_122.gz: (0x00000009) exists archived, archive_id:13
R1_123.gz: (0x00000009) exists archived, archive_id:13
R1_124.gz: (0x00000009) exists archived, archive_id:13
R1_125.gz: (0x00000009) exists archived, archive_id:13
R1_126.gz: (0x00000009) exists archived, archive_id:13
R1_127.gz: (0x00000009) exists archived, archive_id:13
R1_128.gz: (0x00000009) exists archived, archive_id:13
R1_129.gz: (0x00000009) exists archived, archive_id:13
R1_12.gz: (0x00000009) exists archived, archive_id:13
R1_130.gz: (0x00000009) exists archived, archive_id:13
R1_131.gz: (0x00000009) exists archived, archive_id:13
R1_132.gz: (0x00000009) exists archived, archive_id:13
R1_133.gz: (0x00000009) exists archived, archive_id:13
R1_134.gz: (0x00000009) exists archived, archive_id:13
R1_135.gz: (0x00000009) exists archived, archive_id:13
R1_136.gz: (0x00000009) exists archived, archive_id:13
R1_137.gz: (0x00000009) exists archived, archive_id:13
R1_138.gz: (0x00000009) exists archived, archive_id:13
R1_139.gz: (0x00000009) exists archived, archive_id:13
R1_13.gz: (0x00000009) exists archived, archive_id:13
R1_140.gz: (0x00000009) exists archived, archive_id:13
R1_141.gz: (0x00000009) exists archived, archive_id:13
R1_142.gz: (0x00000009) exists archived, archive_id:13
R1_143.gz: (0x00000009) exists archived, archive_id:13
R1_144.gz: (0x00000009) exists archived, archive_id:13
R1_145.gz: (0x00000009) exists archived, archive_id:13
R1_146.gz: (0x00000009) exists archived, archive_id:13
R1_147.gz: (0x00000009) exists archived, archive_id:13
R1_148.gz: (0x00000009) exists archived, archive_id:13
R1_149.gz: (0x00000009) exists archived, archive_id:13
R1_14.gz: (0x00000009) exists archived, archive_id:13
R1_150.gz: (0x00000009) exists archived, archive_id:13
R1_151.gz: (0x00000009) exists archived, archive_id:13
R1_152.gz: (0x00000009) exists archived, archive_id:13
R1_153.gz: (0x00000009) exists archived, archive_id:13
R1_154.gz: (0x00000009) exists archived, archive_id:13
R1_155.gz: (0x00000009) exists archived, archive_id:13
R1_156.gz: (0x00000009) exists archived, archive_id:13
R1_157.gz: (0x00000009) exists archived, archive_id:13
R1_158.gz: (0x00000009) exists archived, archive_id:13
R1_159.gz: (0x00000009) exists archived, archive_id:13
R1_15.gz: (0x00000009) exists archived, archive_id:13
R1_160.gz: (0x00000009) exists archived, archive_id:13
R1_161.gz: (0x00000009) exists archived, archive_id:13
R1_162.gz: (0x00000009) exists archived, archive_id:13
R1_163.gz: (0x00000009) exists archived, archive_id:13
R1_164.gz: (0x00000009) exists archived, archive_id:13
R1_165.gz: (0x00000009) exists archived, archive_id:13
R1_166.gz: (0x00000009) exists archived, archive_id:13
R1_167.gz: (0x00000009) exists archived, archive_id:13
R1_168.gz: (0x00000009) exists archived, archive_id:13
R1_169.gz: (0x00000009) exists archived, archive_id:13
R1_16.gz: (0x00000009) exists archived, archive_id:13
R1_170.gz: (0x00000009) exists archived, archive_id:13
R1_171.gz: (0x00000009) exists archived, archive_id:13
R1_172.gz: (0x00000009) exists archived, archive_id:13
R1_173.gz: (0x00000009) exists archived, archive_id:13
R1_174.gz: (0x00000009) exists archived, archive_id:13
R1_175.gz: (0x00000009) exists archived, archive_id:13
R1_176.gz: (0x00000009) exists archived, archive_id:13
R1_177.gz: (0x00000009) exists archived, archive_id:13
R1_178.gz: (0x00000009) exists archived, archive_id:13
R1_179.gz: (0x00000009) exists archived, archive_id:13
R1_17.gz: (0x00000009) exists archived, archive_id:13
R1_180.gz: (0x00000009) exists archived, archive_id:13
R1_181.gz: (0x00000009) exists archived, archive_id:13
R1_182.gz: (0x00000009) exists archived, archive_id:13
R1_183.gz: (0x00000009) exists archived, archive_id:13
R1_184.gz: (0x00000009) exists archived, archive_id:13
R1_185.gz: (0x00000009) exists archived, archive_id:13
R1_186.gz: (0x00000009) exists archived, archive_id:13
R1_187.gz: (0x00000009) exists archived, archive_id:13
R1_188.gz: (0x00000009) exists archived, archive_id:13
R1_189.gz: (0x00000009) exists archived, archive_id:13
R1_18.gz: (0x00000009) exists archived, archive_id:13
R1_190.gz: (0x00000009) exists archived, archive_id:13
R1_191.gz: (0x00000009) exists archived, archive_id:13
R1_192.gz: (0x00000009) exists archived, archive_id:13
R1_193.gz: (0x00000009) exists archived, archive_id:13
R1_194.gz: (0x00000009) exists archived, archive_id:13
R1_195.gz: (0x00000009) exists archived, archive_id:13
R1_196.gz: (0x00000009) exists archived, archive_id:13
R1_197.gz: (0x00000009) exists archived, archive_id:13
R1_198.gz: (0x00000009) exists archived, archive_id:13
R1_199.gz: (0x00000009) exists archived, archive_id:13
R1_19.gz: (0x00000009) exists archived, archive_id:13
R1_1.gz: (0x00000009) exists archived, archive_id:13
R1_200.gz: (0x00000009) exists archived, archive_id:13
R1_201.gz: (0x00000009) exists archived, archive_id:13
R1_202.gz: (0x00000009) exists archived, archive_id:13
R1_203.gz: (0x00000009) exists archived, archive_id:13
R1_204.gz: (0x00000009) exists archived, archive_id:13
R1_205.gz: (0x00000009) exists archived, archive_id:13
R1_206.gz: (0x00000009) exists archived, archive_id:13
R1_207.gz: (0x00000009) exists archived, archive_id:13
R1_208.gz: (0x00000009) exists archived, archive_id:13
R1_209.gz: (0x00000009) exists archived, archive_id:13
R1_20.gz: (0x00000009) exists archived, archive_id:13
R1_210.gz: (0x00000009) exists archived, archive_id:13
R1_211.gz: (0x00000009) exists archived, archive_id:13
R1_212.gz: (0x00000009) exists archived, archive_id:13
R1_213.gz: (0x00000009) exists archived, archive_id:13
R1_214.gz: (0x00000009) exists archived, archive_id:13
R1_215.gz: (0x00000009) exists archived, archive_id:13
R1_216.gz: (0x00000009) exists archived, archive_id:13
R1_217.gz: (0x00000009) exists archived, archive_id:13
R1_218.gz: (0x00000009) exists archived, archive_id:13
R1_219.gz: (0x00000009) exists archived, archive_id:13
R1_21.gz: (0x00000009) exists archived, archive_id:13
R1_220.gz: (0x00000009) exists archived, archive_id:13
R1_221.gz: (0x00000009) exists archived, archive_id:13
R1_222.gz: (0x00000009) exists archived, archive_id:13
R1_223.gz: (0x00000009) exists archived, archive_id:13
R1_224.gz: (0x00000009) exists archived, archive_id:13
R1_225.gz: (0x00000009) exists archived, archive_id:13
R1_226.gz: (0x00000009) exists archived, archive_id:13
R1_227.gz: (0x00000009) exists archived, archive_id:13
R1_228.gz: (0x00000009) exists archived, archive_id:13
R1_229.gz: (0x00000009) exists archived, archive_id:13
R1_22.gz: (0x00000009) exists archived, archive_id:13
R1_230.gz: (0x00000009) exists archived, archive_id:13
R1_231.gz: (0x00000009) exists archived, archive_id:13
R1_232.gz: (0x00000009) exists archived, archive_id:13
R1_233.gz: (0x00000009) exists archived, archive_id:13
R1_234.gz: (0x00000009) exists archived, archive_id:13
R1_235.gz: (0x00000009) exists archived, archive_id:13
R1_236.gz: (0x00000009) exists archived, archive_id:13
R1_237.gz: (0x00000009) exists archived, archive_id:13
R1_238.gz: (0x00000009) exists archived, archive_id:13
R1_239.gz: (0x00000009) exists archived, archive_id:13
R1_23.gz: (0x00000009) exists archived, archive_id:13
R1_240.gz: (0x00000009) exists archived, archive_id:13
R1_241.gz: (0x00000009) exists archived, archive_id:13
R1_242.gz: (0x00000009) exists archived, archive_id:13
R1_243.gz: (0x00000009) exists archived, archive_id:13
R1_244.gz: (0x00000009) exists archived, archive_id:13
R1_245.gz: (0x00000009) exists archived, archive_id:13
R1_246.gz: (0x00000009) exists archived, archive_id:13
R1_247.gz: (0x00000009) exists archived, archive_id:13
R1_248.gz: (0x00000009) exists archived, archive_id:13
R1_249.gz: (0x00000009) exists archived, archive_id:13
R1_24.gz: (0x00000009) exists archived, archive_id:13
R1_250.gz: (0x00000009) exists archived, archive_id:13
R1_251.gz: (0x00000009) exists archived, archive_id:13
R1_252.gz: (0x00000009) exists archived, archive_id:13
R1_253.gz: (0x00000009) exists archived, archive_id:13
R1_254.gz: (0x00000009) exists archived, archive_id:13
R1_255.gz: (0x00000009) exists archived, archive_id:13
R1_256.gz: (0x00000009) exists archived, archive_id:13
R1_257.gz: (0x00000009) exists archived, archive_id:13
R1_258.gz: (0x00000009) exists archived, archive_id:13
R1_259.gz: (0x00000009) exists archived, archive_id:13
R1_25.gz: (0x00000009) exists archived, archive_id:13
R1_260.gz: (0x00000009) exists archived, archive_id:13
R1_261.gz: (0x00000009) exists archived, archive_id:13
R1_262.gz: (0x00000009) exists archived, archive_id:13
R1_263.gz: (0x00000009) exists archived, archive_id:13
R1_264.gz: (0x00000009) exists archived, archive_id:13
R1_265.gz: (0x00000009) exists archived, archive_id:13
R1_266.gz: (0x00000009) exists archived, archive_id:13
R1_267.gz: (0x00000009) exists archived, archive_id:13
R1_268.gz: (0x00000009) exists archived, archive_id:13
R1_269.gz: (0x00000009) exists archived, archive_id:13
R1_26.gz: (0x00000009) exists archived, archive_id:13
R1_270.gz: (0x00000009) exists archived, archive_id:13
R1_271.gz: (0x00000009) exists archived, archive_id:13
R1_272.gz: (0x00000009) exists archived, archive_id:13
R1_273.gz: (0x00000009) exists archived, archive_id:13
R1_274.gz: (0x00000009) exists archived, archive_id:13
R1_275.gz: (0x00000009) exists archived, archive_id:13
R1_276.gz: (0x00000009) exists archived, archive_id:13
R1_277.gz: (0x00000009) exists archived, archive_id:13
R1_278.gz: (0x00000009) exists archived, archive_id:13
R1_279.gz: (0x00000009) exists archived, archive_id:13
R1_27.gz: (0x00000009) exists archived, archive_id:13
R1_280.gz: (0x00000009) exists archived, archive_id:13
R1_281.gz: (0x00000009) exists archived, archive_id:13
R1_282.gz: (0x00000009) exists archived, archive_id:13
R1_283.gz: (0x00000009) exists archived, archive_id:13
R1_284.gz: (0x00000009) exists archived, archive_id:13
R1_285.gz: (0x00000009) exists archived, archive_id:13
R1_286.gz: (0x00000009) exists archived, archive_id:13
R1_287.gz: (0x00000009) exists archived, archive_id:13
R1_288.gz: (0x00000009) exists archived, archive_id:13
R1_289.gz: (0x00000009) exists archived, archive_id:13
R1_28.gz: (0x00000009) exists archived, archive_id:13
R1_290.gz: (0x00000009) exists archived, archive_id:13
R1_291.gz: (0x00000009) exists archived, archive_id:13
R1_292.gz: (0x00000009) exists archived, archive_id:13
R1_293.gz: (0x00000009) exists archived, archive_id:13
R1_294.gz: (0x00000009) exists archived, archive_id:13
R1_295.gz: (0x00000009) exists archived, archive_id:13
R1_296.gz: (0x00000009) exists archived, archive_id:13
R1_297.gz: (0x00000009) exists archived, archive_id:13
R1_298.gz: (0x00000009) exists archived, archive_id:13
R1_299.gz: (0x00000009) exists archived, archive_id:13
R1_29.gz: (0x00000009) exists archived, archive_id:13
R1_2.gz: (0x00000009) exists archived, archive_id:13
R1_300.gz: (0x00000009) exists archived, archive_id:13
R1_301.gz: (0x00000009) exists archived, archive_id:13
R1_302.gz: (0x00000009) exists archived, archive_id:13
R1_303.gz: (0x00000009) exists archived, archive_id:13
R1_304.gz: (0x00000009) exists archived, archive_id:13
R1_305.gz: (0x00000009) exists archived, archive_id:13
R1_306.gz: (0x00000009) exists archived, archive_id:13
R1_307.gz: (0x00000009) exists archived, archive_id:13
R1_308.gz: (0x00000009) exists archived, archive_id:13
R1_309.gz: (0x00000009) exists archived, archive_id:13
R1_30.gz: (0x00000009) exists archived, archive_id:13
R1_310.gz: (0x00000009) exists archived, archive_id:13
R1_311.gz: (0x00000009) exists archived, archive_id:13
R1_312.gz: (0x00000009) exists archived, archive_id:13
R1_313.gz: (0x00000009) exists archived, archive_id:13
R1_314.gz: (0x00000009) exists archived, archive_id:13
R1_315.gz: (0x00000009) exists archived, archive_id:13
R1_316.gz: (0x00000009) exists archived, archive_id:13
R1_317.gz: (0x00000009) exists archived, archive_id:13
R1_318.gz: (0x00000009) exists archived, archive_id:13
R1_319.gz: (0x00000009) exists archived, archive_id:13
R1_31.gz: (0x00000009) exists archived, archive_id:13
R1_320.gz: (0x00000009) exists archived, archive_id:13
R1_321.gz: (0x00000009) exists archived, archive_id:13
R1_322.gz: (0x00000009) exists archived, archive_id:13
R1_323.gz: (0x00000009) exists archived, archive_id:13
R1_324.gz: (0x00000009) exists archived, archive_id:13
R1_325.gz: (0x00000009) exists archived, archive_id:13
R1_326.gz: (0x00000009) exists archived, archive_id:13
R1_327.gz: (0x00000009) exists archived, archive_id:13
R1_328.gz: (0x00000009) exists archived, archive_id:13
R1_329.gz: (0x00000009) exists archived, archive_id:13
R1_32.gz: (0x00000009) exists archived, archive_id:13
R1_330.gz: (0x00000009) exists archived, archive_id:13
R1_331.gz: (0x00000009) exists archived, archive_id:13
R1_332.gz: (0x00000009) exists archived, archive_id:13
R1_333.gz: (0x00000009) exists archived, archive_id:13
R1_334.gz: (0x00000009) exists archived, archive_id:13
R1_335.gz: (0x00000009) exists archived, archive_id:13
R1_336.gz: (0x00000009) exists archived, archive_id:13
R1_337.gz: (0x00000009) exists archived, archive_id:13
R1_338.gz: (0x00000009) exists archived, archive_id:13
R1_339.gz: (0x00000009) exists archived, archive_id:13
R1_33.gz: (0x00000009) exists archived, archive_id:13
R1_340.gz: (0x00000009) exists archived, archive_id:13
R1_341.gz: (0x00000009) exists archived, archive_id:13
R1_342.gz: (0x00000009) exists archived, archive_id:13
R1_343.gz: (0x00000009) exists archived, archive_id:13
R1_344.gz: (0x00000009) exists archived, archive_id:13
R1_345.gz: (0x00000009) exists archived, archive_id:13
R1_346.gz: (0x00000009) exists archived, archive_id:13
R1_347.gz: (0x00000009) exists archived, archive_id:13
R1_348.gz: (0x00000009) exists archived, archive_id:13
R1_349.gz: (0x00000009) exists archived, archive_id:13
R1_34.gz: (0x00000009) exists archived, archive_id:13
R1_350.gz: (0x00000009) exists archived, archive_id:13
R1_351.gz: (0x00000009) exists archived, archive_id:13
R1_352.gz: (0x00000009) exists archived, archive_id:13
R1_353.gz: (0x00000009) exists archived, archive_id:13
R1_354.gz: (0x00000009) exists archived, archive_id:13
R1_355.gz: (0x00000009) exists archived, archive_id:13
R1_356.gz: (0x00000009) exists archived, archive_id:13
R1_357.gz: (0x00000009) exists archived, archive_id:13
R1_358.gz: (0x00000009) exists archived, archive_id:13
R1_359.gz: (0x00000009) exists archived, archive_id:13
R1_35.gz: (0x00000009) exists archived, archive_id:13
R1_360.gz: (0x00000009) exists archived, archive_id:13
R1_361.gz: (0x00000009) exists archived, archive_id:13
R1_362.gz: (0x00000009) exists archived, archive_id:13
R1_363.gz: (0x00000009) exists archived, archive_id:13
R1_364.gz: (0x00000009) exists archived, archive_id:13
R1_365.gz: (0x00000009) exists archived, archive_id:13
R1_366.gz: (0x00000009) exists archived, archive_id:13
R1_367.gz: (0x00000009) exists archived, archive_id:13
R1_368.gz: (0x00000009) exists archived, archive_id:13
R1_369.gz: (0x00000009) exists archived, archive_id:13
R1_36.gz: (0x00000009) exists archived, archive_id:13
... 2178 more lines ...
  480   Mon Jun 13 16:38:37 2022 NHMon 13 Jun
Remove Caen HV Supply

Open up, C23 was grounded (HV grounded in NIM)
Remove C23 (HV floating at NIM) and reinstall

Rebias HV... No Change

Reboot AIDA... No change
  479   Fri Jun 10 10:32:46 2022 NHFri 10 Jun
Probe PSU output with 'scope again and 10X probe

Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)

Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled 
          Lots of fluctuation and jitter

Figs 4-6: DC 1MOhm output from all 3 power rails 

Fig 7-8: Small "loop" using probe and observed scope spectra.

After lunch ideas:

- Try again with 1X probe attenuation, maybe record output to USB 

- Remove unconnected MACBs from master

- 
Attachment 1: scopePSU1.jpg
scopePSU1.jpg
Attachment 2: scopePSU2.jpg
scopePSU2.jpg
Attachment 3: scoepPSU3.jpg
scoepPSU3.jpg
Attachment 4: scope5V_DC.jpg
scope5V_DC.jpg
Attachment 5: scopeN6V_DC.jpg
scopeN6V_DC.jpg
Attachment 6: scope_7VDC.jpg
scope_7VDC.jpg
Attachment 7: scope_RFAnt.jpg
scope_RFAnt.jpg
Attachment 8: scopeRF_pickup.jpg
scopeRF_pickup.jpg
  478   Wed Jun 8 09:55:28 2022 NHWeds 8 May
10.30 : Turn off AIDA FEE64s 

        Prepare a ground cable for snout - M10 terminal on one end for copper bar. Will affix with Cu tape for now (no free screws)

Plan for today:

Remove extraneous ground cables
Check ground of all FEE64s esp. 05 and 07 - include on FEE64 side.
 14:42 - Done all grounds seem snug, extra ground cables removed. Try power on and check

Try powering just DSSD1 FEE64s
 - No change (aida01 125k, aida02,03,04 - 400k)

Try grounding snout
 14:59 Connection to snout very bad as tape does not fix well to thick copper in braid. nO difference either way.

Probe PSU output with Battery DSO if available
 - See figure 2

Try shuffling FEE64 power allocation
 - 15:32 - Now as 1 X 3 X 2 X 4 X
                  5 X 7 X 6 X 8 X
(Also FEEs on own pair)
   No difference to rates

Multimeter voltages for last socket of PSU1 (paired with aida04):
  1 : 6.02 V
  2:  6.02 V
  3:  6.02 V
  4: -6.44 V
  5:  7.74 V

Similar on PSU2

Voltage between right (GND) pin 5 and 1 (pin 5 is COM) is -0.4 V, similar if pin 4 is COM. Pin1-3 are 0 V when paired
  
Attachment 1: 2022-06-08_14-48-21_goodevents.png
2022-06-08_14-48-21_goodevents.png
Attachment 2: rn_image_picker_lib_temp_97373248-bdea-4864-b8a9-a13d0d016e30.jpg
rn_image_picker_lib_temp_97373248-bdea-4864-b8a9-a13d0d016e30.jpg
  477   Wed Jun 1 08:00:18 2022 TD2xDSSSDs spectra layouts
Spectra layouts for 2x DSSSDs can be found in directory ~/LayOut/2xDSSSDs
  476   Wed Jun 1 07:35:13 2022 TDWednesday 1 June
Attachment 1 - DSSSD bias & leakage current - OK

Attachment 2 - ADC data item stats

Attachment 3 - FEE64 temps - OK

Attachments 4-7 - aida01, 02, 03 & 04 2.*.W spectra

Noise observed for asics #1 & #2 of aida02 consistent with no cable attached - confirmed by visual inspection - ribbon cable disconnected from Samtec header of adaptor PCB

To Do list

1) re-connect ribbon cable to aida02 adaptor PCB
2) good quality ground to snout - use screws at inter-stage
3) install heavy duty grounds
4) ... 


Attachment 8 - adc data item stats - slow comparator 0x64

Attachment 9 - per FEE64 rate spectra - slow comparator 0x64

Attachment 10 - per FEE64 rate spectra - slow comparator 0xa

-

13:30 - Attach heavy duty ground to AIDA 

Attachment 11 - per FEE64 rate spectra - slow comparator 0xa

Try power cycle as system ground changed significantly now

Try to reattach aida04 TTY connector - still wrong

Archive old tty logs (ttyUSB*) to zip and delete them, makes easier to track startup 

Attachment 12 - per FEE64 rate spectra - slow comparator 0xa

No major difference, aida01 quieter than before

aida05 has some missing channels - loose ERNI?

Attachements 13 & 14 - Waveforms

Temporarily raise to 0x64 and test merger for DTAS synchro test later... all works good to MBS 
Change NETVAR MERGE.LinksAvailable to 8

Notice one copper screw touching aluminium, add some kapton tape behind it to isolate it. No difference

Voltage diff between aida02,aida06 < 1 mV
Voltage diff between copper bars < 0.5 mV (but not 0)

A comment from DTAS people that they observed a large increase in noise recently - perhaps from accelerator
May indicate current power/gronud oddities?

Attachment 15 - Rates at 0x1f (= 310 keV)

DSSSD#1 reasonable (aida04 a bit bad), but mostly just pulser (25 Hz)

DSSSD#2 much worse esp. p+n sides (aida05, aida07)

ERNI connectors pushed in and feel fully in
Attachment 1: Screenshot_from_2022-06-01_08-33-30.png
Screenshot_from_2022-06-01_08-33-30.png
Attachment 2: Screenshot_from_2022-06-01_08-33-13.png
Screenshot_from_2022-06-01_08-33-13.png
Attachment 3: Screenshot_from_2022-06-01_08-32-44.png
Screenshot_from_2022-06-01_08-32-44.png
Attachment 4: Screenshot_from_2022-06-01_08-32-14.png
Screenshot_from_2022-06-01_08-32-14.png
Attachment 5: Screenshot_from_2022-06-01_08-31-04.png
Screenshot_from_2022-06-01_08-31-04.png
Attachment 6: Screenshot_from_2022-06-01_08-29-37.png
Screenshot_from_2022-06-01_08-29-37.png
Attachment 7: Screenshot_from_2022-06-01_08-27-41.png
Screenshot_from_2022-06-01_08-27-41.png
Attachment 8: Screenshot_from_2022-06-01_08-54-54.png
Screenshot_from_2022-06-01_08-54-54.png
Attachment 9: Screenshot_from_2022-06-01_08-54-34.png
Screenshot_from_2022-06-01_08-54-34.png
Attachment 10: Screenshot_from_2022-06-01_08-51-51.png
Screenshot_from_2022-06-01_08-51-51.png
Attachment 11: 2022-06-01_13-29-47_stats.png
2022-06-01_13-29-47_stats.png
Attachment 12: 2022-06-01_13-42-07_rates.png
2022-06-01_13-42-07_rates.png
Attachment 13: 2022-06-01_13-44-25_waves_pn.png
2022-06-01_13-44-25_waves_pn.png
Attachment 14: 2022-06-01_13-44-52_nn_waves.png
2022-06-01_13-44-52_nn_waves.png
Attachment 15: 2022-06-01_14-23-57_0x1f.png
2022-06-01_14-23-57_0x1f.png
  475   Tue May 31 17:09:35 2022 TDTuesday 31 May
2x MSL type BB18(DS)-1000 installed


Adaptor PCB LK config

all FEE64s LK2, LK4 fitted
n+n FEE64s LK1 fitted
bottom p+n FEE64s LK3 fitted


AIDA PSU config

top 1-3 2-4
bottom 5-7 6-8


DSSSD bias -> p+n FEE64 core, n+n FEE64 braid


Heavy duty grounds not connected to FEE64 adaptor PCBs


ASIC settings 2021Apr29-13-16-00
 slow comparator 0xa

BNC PB-5 settings
 amplitude 1.000V
 attenuation x1
 tau_d 1ms
 frequency 25Hz
 polarity +


Attachment 1 - DSSSD bias & Leakage current - OK

Attachment 2 - FEE64 temps OK

Attachment 3 - adc data item stats

Attachments 4-9 - system wide checks OK *except* aida07 fails ADC calibration

Attachments 10 & 11 - 1.8.W spectra 20us FSR

Attachments 12 & 13 - 1.8.L spectra - aida03 pulser peak width 155 ch FWHM

Attachment 14 - per FEE64 rate spectra



19.15 Attachments 15-18 - 2*W spectra for aida01, 02, 03 &04

Attachment 19 - per FEE64 rate spectra

Attachment 20 - adc data item stats
Attachment 1: Screenshot_from_2022-05-31_18-09-27.png
Screenshot_from_2022-05-31_18-09-27.png
Attachment 2: Screenshot_from_2022-05-31_18-09-49.png
Screenshot_from_2022-05-31_18-09-49.png
Attachment 3: Screenshot_from_2022-05-31_18-10-18.png
Screenshot_from_2022-05-31_18-10-18.png
Attachment 4: Screenshot_from_2022-05-31_18-17-27.png
Screenshot_from_2022-05-31_18-17-27.png
Attachment 5: Screenshot_from_2022-05-31_18-17-02.png
Screenshot_from_2022-05-31_18-17-02.png
Attachment 6: Screenshot_from_2022-05-31_18-16-43.png
Screenshot_from_2022-05-31_18-16-43.png
Attachment 7: Screenshot_from_2022-05-31_18-16-28.png
Screenshot_from_2022-05-31_18-16-28.png
Attachment 8: Screenshot_from_2022-05-31_18-16-12.png
Screenshot_from_2022-05-31_18-16-12.png
Attachment 9: Screenshot_from_2022-05-31_18-15-13.png
Screenshot_from_2022-05-31_18-15-13.png
Attachment 10: Screenshot_from_2022-05-31_18-32-34.png
Screenshot_from_2022-05-31_18-32-34.png
Attachment 11: Screenshot_from_2022-05-31_18-31-56.png
Screenshot_from_2022-05-31_18-31-56.png
Attachment 12: Screenshot_from_2022-05-31_18-30-55.png
Screenshot_from_2022-05-31_18-30-55.png
Attachment 13: Screenshot_from_2022-05-31_18-29-53.png
Screenshot_from_2022-05-31_18-29-53.png
Attachment 14: Screenshot_from_2022-05-31_18-28-56.png
Screenshot_from_2022-05-31_18-28-56.png
Attachment 15: Screenshot_from_2022-05-31_19-13-51.png
Screenshot_from_2022-05-31_19-13-51.png
Attachment 16: Screenshot_from_2022-05-31_19-13-03.png
Screenshot_from_2022-05-31_19-13-03.png
Attachment 17: Screenshot_from_2022-05-31_19-12-19.png
Screenshot_from_2022-05-31_19-12-19.png
Attachment 18: Screenshot_from_2022-05-31_19-11-24.png
Screenshot_from_2022-05-31_19-11-24.png
Attachment 19: Screenshot_from_2022-05-31_19-09-10.png
Screenshot_from_2022-05-31_19-09-10.png
Attachment 20: Screenshot_from_2022-05-31_19-06-59.png
Screenshot_from_2022-05-31_19-06-59.png
  474   Tue May 31 09:05:00 2022 NH, TDTuesday 31 May
DSSD stack will be as follows

1      3208-7  1019       125              upstream
2      3208-13 1020       120              downstream

First start checking bias pins on both detectors
These are pins on the *bottom right* of each 68 pin connector (C-2299)
Expect finite resistance

3208-7: 
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

3208-13:
  Top and bottom: 22 Ohm
  Left and right: 22 Ohm

Conclusion: All bias bond wires are intact on both DSSDs

Visual inspection is both DSSDs look good with no damage or bond wire issues

Installed in snout: full stack

3 mm bPlas 1
17 mm Air
1 mm AIDA 1  (645 mm from base of snout)
9 mm Air
1 mm AIDA 2 (655 mm from base of snout)
12 mm Air
3 mm bPlas 2
  473   Wed May 25 14:28:26 2022 NHWed 25 May
The n+n FEEs were all removed to adjust the platform back to the single configuration

After some confusion we found the orientation of the frame to position the single detectors and moved back

Faulty AIDA cards aida05 and aida14 were removed to be returned backi (ASIC damage)

The cables for aida10 and aida12 were moved back to the left-right side and the n+n cards were installed and reocnnected

p+n cards moved to bring aida11 and aida09 above aida03/aida07 and aida01/aida05. Also move a different card to replace old aida05

While doing this it was noticed the HDMI for aida13 was very loose and actually fell off (I guess the cable was pulled incorrectly)
It has been removed and will eb sent back to be resoldered 

The copper grounding bars and ITEM frame were removed due to conflicting with the single snout layout
The bolts are slightly too short to hold the frame where it won't conflict so we loko for longer bolts to reattach the system
Some new ground cables will be made to reach the extra n+n FEEs

Aida13-Aida16 removed from the PSU 

fig1: layout of snout frame in "single" mode (alu block configuration)


MAC Addresses

aida01 - ba-8a
aida05 - 0d:15
aida09 - ee-01

aida04 - f6-b7
aida08 - ba-2b
aida12 - ba:89 

aida03 - d8-21
aida07 - 41:b4:0c
aida11 - f6-5a

aida02 - ba:22
aida06 - ee:72
aida10 - cf-ac [mistakenly swapped with aida12]

Second switch will stay in S4 and has now been screwed in properly below the switch 1 and the hdmi cables entrance

DHCP *not* updated

Relay is powered *off* so FEEs cannot be turned on, alL HV channels disabled

Tomorrow is a holiday so no work will be happening @ GSI
Attachment 1: aida_single_alu.jpg
aida_single_alu.jpg
  472   Tue May 24 08:35:00 2022 NHTue 24 May
09:00 - Remove bias of DSSDs  and pwoer of FEEs

Remove snout from S4 so DEGAS frame can be mechnically removed

Snout now in FRS Messhuette 
  471   Mon May 23 16:41:45 2022 TDMonday 23 May
17.40 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-23_17-39-23.png
Screenshot_from_2022-05-23_17-39-23.png
Attachment 2: Screenshot_from_2022-05-23_17-39-58.png
Screenshot_from_2022-05-23_17-39-58.png
Attachment 3: Screenshot_from_2022-05-23_17-40-33.png
Screenshot_from_2022-05-23_17-40-33.png
  470   Sun May 22 14:12:55 2022 TDSunday 22 May
15.11 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-22_15-10-25.png
Screenshot_from_2022-05-22_15-10-25.png
Attachment 2: Screenshot_from_2022-05-22_15-10-48.png
Screenshot_from_2022-05-22_15-10-48.png
Attachment 3: Screenshot_from_2022-05-22_15-11-35.png
Screenshot_from_2022-05-22_15-11-35.png
  469   Sat May 21 15:30:45 2022 TDSaturday 21 May
16.31 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current - OK, temp data has ceased

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-21_16-27-58.png
Screenshot_from_2022-05-21_16-27-58.png
Attachment 2: Screenshot_from_2022-05-21_16-28-59.png
Screenshot_from_2022-05-21_16-28-59.png
Attachment 3: Screenshot_from_2022-05-21_16-29-36.png
Screenshot_from_2022-05-21_16-29-36.png
  468   Fri May 20 16:22:01 2022 TDFriday 20 May
17.20 DAQ stopped

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps - OK
Attachment 1: Screenshot_from_2022-05-20_17-19-21.png
Screenshot_from_2022-05-20_17-19-21.png
Attachment 2: Screenshot_from_2022-05-20_17-19-35.png
Screenshot_from_2022-05-20_17-19-35.png
Attachment 3: Screenshot_from_2022-05-20_17-20-22.png
Screenshot_from_2022-05-20_17-20-22.png
  467   Thu May 19 17:00:46 2022 TDThursday 19 May
18.01 DAQ stopped

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachment 2 - DSSSD bias & leakage current

Attachment 3 - FEE64 temps - OK

Attachment 4 - ADC data item stats
Attachment 1: Screenshot_from_2022-05-19_17-59-41.png
Screenshot_from_2022-05-19_17-59-41.png
Attachment 2: Screenshot_from_2022-05-19_17-58-21.png
Screenshot_from_2022-05-19_17-58-21.png
Attachment 3: Screenshot_from_2022-05-19_17-58-52.png
Screenshot_from_2022-05-19_17-58-52.png
Attachment 4: Screenshot_from_2022-05-19_17-59-20.png
Screenshot_from_2022-05-19_17-59-20.png
  466   Wed May 18 15:29:50 2022 TDWednesday 18 May
      file no storage mode, no MBS data transfer

      ADC control register 0x0 

      std disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0xa

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Attachment 1 - DSSSD bias & leakage current

Attachment 2 - grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - ADC data item stats

Attachment 4 - FEE64 temps - OK

16.35 zero all histograms
      zero statistics
      zero NewMerger statistics

      ASIC check control - all FEE64s, all ASICs

      baseline counters
      system wide checks - all OK *except* aida07 WR error 0x10

Attachment 5 - per FEE64 rate spectra

Attachmnents 6 & 7 - per FEE64 1.8.W spectra 20us FSR
Attachment 1: Screenshot_from_2022-05-18_16-24-18.png
Screenshot_from_2022-05-18_16-24-18.png
Attachment 2: Screenshot_from_2022-05-18_16-24-41.png
Screenshot_from_2022-05-18_16-24-41.png
Attachment 3: Screenshot_from_2022-05-18_16-26-13.png
Screenshot_from_2022-05-18_16-26-13.png
Attachment 4: Screenshot_from_2022-05-18_16-26-53.png
Screenshot_from_2022-05-18_16-26-53.png
Attachment 5: Screenshot_from_2022-05-18_16-36-47.png
Screenshot_from_2022-05-18_16-36-47.png
Attachment 6: Screenshot_from_2022-05-18_16-39-48.png
Screenshot_from_2022-05-18_16-39-48.png
Attachment 7: Screenshot_from_2022-05-18_16-52-18.png
Screenshot_from_2022-05-18_16-52-18.png
Attachment 8: Screenshot_from_2022-05-18_16-53-32.png
Screenshot_from_2022-05-18_16-53-32.png
  465   Tue May 17 16:53:20 2022 TDTuesday 17 May 16.00-00.00
17.35 zero all histograms
      zero statistics
      zero NewMerger statistics

      ASIC check control - all FEE64s, all ASICs

      file no storage mode, no MBS data transfer

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0x80

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachments 2-10 - NewMerger stats

Attachments 11-13 - iptraf, TapeServer, NewMerger

Attachment 14 - per Fee64 rate spectra

Attachment 15 - lost activity  monitor

Attachment 16 - ASIC settings aida01

Attachments 17-22 - adc, pause, resume & correlation scaler data items, push, flush, aida01 & aida06 stats

Attachment 23 - FEE64 temperatures OK

Attachment 24 - DSSSD bias & leakage current
Attachment 1: Screenshot_from_2022-05-17_17-52-15.png
Screenshot_from_2022-05-17_17-52-15.png
Attachment 2: Screenshot_from_2022-05-17_17-50-50.png
Screenshot_from_2022-05-17_17-50-50.png
Attachment 3: Screenshot_from_2022-05-17_17-49-18.png
Screenshot_from_2022-05-17_17-49-18.png
Attachment 4: Screenshot_from_2022-05-17_17-48-24.png
Screenshot_from_2022-05-17_17-48-24.png
Attachment 5: Screenshot_from_2022-05-17_17-47-35.png
Screenshot_from_2022-05-17_17-47-35.png
Attachment 6: Screenshot_from_2022-05-17_17-47-12.png
Screenshot_from_2022-05-17_17-47-12.png
Attachment 7: Screenshot_from_2022-05-17_17-46-19.png
Screenshot_from_2022-05-17_17-46-19.png
Attachment 8: Screenshot_from_2022-05-17_17-45-34.png
Screenshot_from_2022-05-17_17-45-34.png
Attachment 9: Screenshot_from_2022-05-17_17-45-10.png
Screenshot_from_2022-05-17_17-45-10.png
Attachment 10: Screenshot_from_2022-05-17_17-44-34.png
Screenshot_from_2022-05-17_17-44-34.png
Attachment 11: Screenshot_from_2022-05-17_17-44-03.png
Screenshot_from_2022-05-17_17-44-03.png
Attachment 12: Screenshot_from_2022-05-17_17-43-37.png
Screenshot_from_2022-05-17_17-43-37.png
Attachment 13: Screenshot_from_2022-05-17_17-42-58.png
Screenshot_from_2022-05-17_17-42-58.png
Attachment 14: Screenshot_from_2022-05-17_17-42-40.png
Screenshot_from_2022-05-17_17-42-40.png
Attachment 15: Screenshot_from_2022-05-17_17-40-53.png
Screenshot_from_2022-05-17_17-40-53.png
Attachment 16: Screenshot_from_2022-05-17_17-40-08.png
Screenshot_from_2022-05-17_17-40-08.png
Attachment 17: Screenshot_from_2022-05-17_17-39-20.png
Screenshot_from_2022-05-17_17-39-20.png
Attachment 18: Screenshot_from_2022-05-17_17-38-52.png
Screenshot_from_2022-05-17_17-38-52.png
Attachment 19: Screenshot_from_2022-05-17_17-38-20.png
Screenshot_from_2022-05-17_17-38-20.png
Attachment 20: Screenshot_from_2022-05-17_17-37-47.png
Screenshot_from_2022-05-17_17-37-47.png
Attachment 21: Screenshot_from_2022-05-17_17-37-19.png
Screenshot_from_2022-05-17_17-37-19.png
Attachment 22: Screenshot_from_2022-05-17_17-36-50.png
Screenshot_from_2022-05-17_17-36-50.png
Attachment 23: Screenshot_from_2022-05-17_17-36-14.png
Screenshot_from_2022-05-17_17-36-14.png
Attachment 24: Screenshot_from_2022-05-17_17-35-42.png
Screenshot_from_2022-05-17_17-35-42.png
  464   Tue May 17 06:58:51 2022 OHTuesday 17th May
08:00 Still have beam for the time being.
      Currently on file R4_2049

      FEE Stats - attachment 1
      FEE Temps ok - attachment 2
      Bias and leakage currents - attachment 3
      System wide checks all ok *except aida09 fails clock check with value 6 which is ok*

      Plan for the morning is once beam is lost is Ge group will enter and place source on the snout to perform efficiency measurement.
      Once they have left the area and are taking data a pulser walkthrough will be performed.
      Patrick can then have access to the system

08:16 Beam taken away
      Data taken stopped on file R4_2055
      Will perform a pulser walkthrough now

      Pulser walkthrough started on R5
      1V->0.1V in 0.1V steps
      Pulser setting - attachment 4
      p+n pulser peaks - attachment 5
      n+n pulser "peaks" - attachment 6
Attachment 1: Screenshot_from_2022-05-17_08-00-50.png
Screenshot_from_2022-05-17_08-00-50.png
Attachment 2: Screenshot_from_2022-05-17_08-01-13.png
Screenshot_from_2022-05-17_08-01-13.png
Attachment 3: Screenshot_from_2022-05-17_08-01-25.png
Screenshot_from_2022-05-17_08-01-25.png
Attachment 4: Screenshot_from_2022-05-17_08-49-02.png
Screenshot_from_2022-05-17_08-49-02.png
Attachment 5: Screenshot_from_2022-05-17_08-51-19.png
Screenshot_from_2022-05-17_08-51-19.png
Attachment 6: Screenshot_from_2022-05-17_08-51-52.png
Screenshot_from_2022-05-17_08-51-52.png
  463   Mon May 16 23:02:54 2022 AMTuesday 17th May 00:00-08:00

01:00:     Attachments 1-3, System checks good

03:00:     Attachments 4-6, System checks good

05:00:     Attachments 7-9, System checks good

07:00:     Attachments 10-12, System checks good

Attachment 1: 220512_0604_biasLeakage.png
220512_0604_biasLeakage.png
Attachment 2: 220512_0603_temps.png
220512_0603_temps.png
Attachment 3: 220512_0601_stats.png
220512_0601_stats.png
Attachment 4: 220517_0300_temps.png
220517_0300_temps.png
Attachment 5: 220517_0300_stats.png
220517_0300_stats.png
Attachment 6: 220517_0300_biasLeakage.png
220517_0300_biasLeakage.png
Attachment 7: 220517_0500_temps.png
220517_0500_temps.png
Attachment 8: 220517_0500_stats.png
220517_0500_stats.png
Attachment 9: 220517_0500_biasLeakage.png
220517_0500_biasLeakage.png
Attachment 10: 220517_0700_temps.png
220517_0700_temps.png
Attachment 11: 220517_0700_stats.png
220517_0700_stats.png
Attachment 12: 220517_0700_biasLeakage.png
220517_0700_biasLeakage.png
  462   Mon May 16 17:23:27 2022 TDNewMerger stats
Attachment 1: Screenshot_from_2022-05-16_18-17-10.png
Screenshot_from_2022-05-16_18-17-10.png
Attachment 2: Screenshot_from_2022-05-16_18-17-50.png
Screenshot_from_2022-05-16_18-17-50.png
Attachment 3: Screenshot_from_2022-05-16_18-18-24.png
Screenshot_from_2022-05-16_18-18-24.png
Attachment 4: Screenshot_from_2022-05-16_18-20-02.png
Screenshot_from_2022-05-16_18-20-02.png
  461   Mon May 16 15:12:50 2022 RDP, MAMonday 16th May 16:00-00:00

16:00 Took over the shift from Magda.

17:30 Updates: Attached (1-3)

19:30 Updates: Attached (4-6)

21:30 Updates: Attached (7-9)

23:00 Updates: Attached (10-12)

 

 

 

Attachment 1: Statistics_2022-05-16_17-32-40.png
Statistics_2022-05-16_17-32-40.png
Attachment 2: Temperature_2022-05-16_17-34-15.png
Temperature_2022-05-16_17-34-15.png
Attachment 3: LeakageCurrent_2022-05-16_17-35-05.png
LeakageCurrent_2022-05-16_17-35-05.png
Attachment 4: Statistics2022-05-16_19-31-28.png
Statistics2022-05-16_19-31-28.png
Attachment 5: Temperature2022-05-16_19-32-39.png
Temperature2022-05-16_19-32-39.png
Attachment 6: LeakageCurrent_2022-05-16_19-30-41.png
LeakageCurrent_2022-05-16_19-30-41.png
Attachment 7: Statistics2022-05-16_21-27-17.png
Statistics2022-05-16_21-27-17.png
Attachment 8: Temperature_2022-05-16_21-28-21.png
Temperature_2022-05-16_21-28-21.png
Attachment 9: LeakageCurrent2022-05-16_21-26-15.png
LeakageCurrent2022-05-16_21-26-15.png
Attachment 10: Statistics2022-05-16_22-59-14.png
Statistics2022-05-16_22-59-14.png
Attachment 11: Temperature_2022-05-16_23-00-18.png
Temperature_2022-05-16_23-00-18.png
Attachment 12: LeakageCurrent_2022-05-16_22-58-06.png
LeakageCurrent_2022-05-16_22-58-06.png
  460   Mon May 16 14:07:06 2022 TDLost activity monitor
Attachment 1: Screenshot_from_2022-05-16_15-04-56.png
Screenshot_from_2022-05-16_15-04-56.png
  459   Mon May 16 07:03:16 2022 MS, OHMonday 16th May 08:00-16:00

08:00 Took over the night shift from Tom (Attachments 1-3)

10:00: Attachement 4-6

around 10:30: Beam went off (Attachment 7)

11:00: Beam is back

12:00 Attachments 8-10

14:00 Attachements 11-13

15:50 Attachments 14-17

Attachment 1: Screenshot_2022-05-16_at_07-59-51_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_07-59-51_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-16_at_08-00-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_08-00-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-16_08-01-20.png
Screenshot_from_2022-05-16_08-01-20.png
Attachment 4: Screenshot_2022-05-16_at_09-59-15_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_09-59-15_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2022-05-16_at_09-59-45_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_09-59-45_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: Screenshot_from_2022-05-16_10-01-47.png
Screenshot_from_2022-05-16_10-01-47.png
Attachment 7: Screenshot_2022-05-16_at_10-47-01_ucesb.png
Screenshot_2022-05-16_at_10-47-01_ucesb.png
Attachment 8: Screenshot_2022-05-16_at_11-59-53_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_11-59-53_Statistics_aidas-gsi.png
Attachment 9: Screenshot_2022-05-16_at_12-00-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_12-00-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 10: Screenshot_from_2022-05-16_12-00-57.png
Screenshot_from_2022-05-16_12-00-57.png
Attachment 11: Screenshot_2022-05-16_at_14-00-56_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_14-00-56_Statistics_aidas-gsi.png
Attachment 12: Screenshot_2022-05-16_at_14-01-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_14-01-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 13: Screenshot_from_2022-05-16_14-01-56.png
Screenshot_from_2022-05-16_14-01-56.png
Attachment 14: Screenshot_2022-05-16_at_15-47-58_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_15-47-58_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2022-05-16_at_15-48-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_15-48-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 16: Screenshot_2022-05-16_at_15-48-34_Lost_activity_monitor_aidas-gsi.png
Screenshot_2022-05-16_at_15-48-34_Lost_activity_monitor_aidas-gsi.png
Attachment 17: Screenshot_from_2022-05-16_15-49-34.png
Screenshot_from_2022-05-16_15-49-34.png
  458   Sun May 15 23:00:08 2022 & TDMonday 16 May 00:00-08:00
00.01 DAQ continues
      file S450/R4_1319

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator aida02 & aida04 0x16, aida06 & aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Disk space OK - /media/SecondDrive

[npg@aidas-gsi S450]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G  389M  7.4G   5% /dev/shm
tmpfs                    7.8G   19M  7.7G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   16G   35G  31% /
/dev/sda2               1014M  226M  789M  23% /boot
/dev/sda1                200M   12M  189M   6% /boot/efi
/dev/sde1                7.2T  4.1T  2.8T  61% /media/SecondDrive
/dev/mapper/centos-home  407G   91G  316G  23% /home
tmpfs                    1.6G   52K  1.6G   1% /run/user/1000
/dev/sdd1                7.2T  6.5T  310G  96% /run/media/npg/ThirdDrive

00.03 all histograms zero'd
      system wide checks counter baseline


00.08 check ASIC control - all FEE64s all ASICs

Attachments 1 & 2 - DSSSD bias & Leakage current - OK
                    grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps OK

Attachments 4-9 - adc, pause, resume & correlation scaler data items, push, flush stats

Attachments 10-16 - TapeServer, NewMerger, NewMerger stats

Attachment 17 - ucesb

04.08

Attachment 18 - DSSSD bias & Leakage current - OK

Attachment 19 - FEE64 temps OK

Attachment 20 - adc data item stats

Attachments 21 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

06.36

Attachment 22 - DSSSD bias & Leakage current - OK

Attachment 23 - FEE64 temps OK

Attachment 24 - adc data item stats

Attachments 25 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6
Attachment 1: Screenshot_from_2022-05-16_00-10-13.png
Screenshot_from_2022-05-16_00-10-13.png
Attachment 2: Screenshot_from_2022-05-16_00-10-43.png
Screenshot_from_2022-05-16_00-10-43.png
Attachment 3: Screenshot_from_2022-05-16_00-11-18.png
Screenshot_from_2022-05-16_00-11-18.png
Attachment 4: Screenshot_from_2022-05-16_00-11-50.png
Screenshot_from_2022-05-16_00-11-50.png
Attachment 5: Screenshot_from_2022-05-16_00-12-27.png
Screenshot_from_2022-05-16_00-12-27.png
Attachment 6: Screenshot_from_2022-05-16_00-12-54.png
Screenshot_from_2022-05-16_00-12-54.png
Attachment 7: Screenshot_from_2022-05-16_00-13-25.png
Screenshot_from_2022-05-16_00-13-25.png
Attachment 8: Screenshot_from_2022-05-16_00-13-55.png
Screenshot_from_2022-05-16_00-13-55.png
Attachment 9: Screenshot_from_2022-05-16_00-14-22.png
Screenshot_from_2022-05-16_00-14-22.png
Attachment 10: Screenshot_from_2022-05-16_00-15-46.png
Screenshot_from_2022-05-16_00-15-46.png
Attachment 11: Screenshot_from_2022-05-16_00-16-08.png
Screenshot_from_2022-05-16_00-16-08.png
Attachment 12: Screenshot_from_2022-05-16_00-16-36.png
Screenshot_from_2022-05-16_00-16-36.png
Attachment 13: Screenshot_from_2022-05-16_00-16-58.png
Screenshot_from_2022-05-16_00-16-58.png
Attachment 14: Screenshot_from_2022-05-16_00-16-58.png
Screenshot_from_2022-05-16_00-16-58.png
Attachment 15: Screenshot_from_2022-05-16_00-17-32.png
Screenshot_from_2022-05-16_00-17-32.png
Attachment 16: Screenshot_from_2022-05-16_00-17-55.png
Screenshot_from_2022-05-16_00-17-55.png
Attachment 17: Screenshot_from_2022-05-16_00-18-45.png
Screenshot_from_2022-05-16_00-18-45.png
Attachment 18: Screenshot_from_2022-05-16_04-08-34.png
Screenshot_from_2022-05-16_04-08-34.png
Attachment 19: Screenshot_from_2022-05-16_04-09-07.png
Screenshot_from_2022-05-16_04-09-07.png
Attachment 20: Screenshot_from_2022-05-16_04-09-57.png
Screenshot_from_2022-05-16_04-09-57.png
Attachment 21: Screenshot_from_2022-05-16_04-10-27.png
Screenshot_from_2022-05-16_04-10-27.png
Attachment 22: Screenshot_from_2022-05-16_06-35-32.png
Screenshot_from_2022-05-16_06-35-32.png
Attachment 23: Screenshot_from_2022-05-16_06-36-37.png
Screenshot_from_2022-05-16_06-36-37.png
Attachment 24: Screenshot_from_2022-05-16_06-37-04.png
Screenshot_from_2022-05-16_06-37-04.png
Attachment 25: Screenshot_from_2022-05-16_06-37-58.png
Screenshot_from_2022-05-16_06-37-58.png
  457   Sun May 15 15:51:49 2022 MLSunday 15th May 16:00-0:00

Status at 16:45 (CET)

The expreiment continues to run smoothly.

AIDA Stats look ok - Attachment 1

AIDA Temperature ok - Attachment 2

AIDA Leakage current: ok - Attachment 3

System wide check:

Clock: 13 passed, 1 failed (aida09)

ADC Calibration: 9 passed, 5 fialed (aida2,6,9,10,13)

White rabbit decoder: 14 passed, 0 failed.

FPGA timestamp: 14 passed, 0 failed.

 

Status at 18:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 4

AIDA Temperature ok - Attachment 5

AIDA Leakage current: increase slightly - Attachment 6

System wide check: No changes, same as above

 

Status at 20:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 7

AIDA Temperature ok - Attachment 8

AIDA Leakage current: increase slightly - Attachment 9

System wide check: No changes, same as above

 

Status at 22:45 (CET):

All rates ok on ucesb.

AIDA Stats ok - Attachment 10

AIDA Temperature ok - Attachment 11

AIDA Leakage current: increase slightly - Attachment 12

System wide check: No changes, same as above

Attachment 1: Stats_Screenshot_from_2022-05-15_16-49-06.png
Stats_Screenshot_from_2022-05-15_16-49-06.png
Attachment 2: Temp_Screenshot_from_2022-05-15_16-48-31.png
Temp_Screenshot_from_2022-05-15_16-48-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-15_16-47-44.png
HV_LC_Screenshot_from_2022-05-15_16-47-44.png
Attachment 4: Stats_Screenshot_from_2022-05-15_18-43-16.png
Stats_Screenshot_from_2022-05-15_18-43-16.png
Attachment 5: Temp_Screenshot_from_2022-05-15_18-44-06.png
Temp_Screenshot_from_2022-05-15_18-44-06.png
Attachment 6: HV_LC_Screenshot_from_2022-05-15_18-42-23.png
HV_LC_Screenshot_from_2022-05-15_18-42-23.png
Attachment 7: Stats_Screenshot_from_2022-05-15_20-42-03.png
Stats_Screenshot_from_2022-05-15_20-42-03.png
Attachment 8: Temp_Screenshot_from_2022-05-15_20-42-42.png
Temp_Screenshot_from_2022-05-15_20-42-42.png
Attachment 9: HV_LC_Screenshot_from_2022-05-15_20-36-28.png
HV_LC_Screenshot_from_2022-05-15_20-36-28.png
Attachment 10: Stats_Screenshot_from_2022-05-15_22-43-37.png
Stats_Screenshot_from_2022-05-15_22-43-37.png
Attachment 11: Temp_Screenshot_from_2022-05-15_22-44-20.png
Temp_Screenshot_from_2022-05-15_22-44-20.png
Attachment 12: HV_LC_Screenshot_from_2022-05-15_22-42-36.png
HV_LC_Screenshot_from_2022-05-15_22-42-36.png
  456   Sun May 15 07:40:52 2022 OHSunday 15th May 08:00-16:00
08:40 FEE Temperatures ok - attachment 1
      FEE statistics - Attachment 2
      Bias and leakage currents ok - attachment 3
      ASIC check ok
      All system wide checks ok - Except aida09 fails clock check with bit 6

11:00 FEE statistics - attachment 4
      FEE temperatures all ok - attachment 5
      BIAS and leakage currents - attachment 6
      ASIC check ok
      All system wide checks as before
      Currently on file R4_1030
      13.1 MB/s to disk
      4.8 million items per second
      
13:13 FEE statistics - attachment 7
      FEE Temperatures all ok - attahcment 8
      Bias and leakage currents ok - attachment 9
      ASIC check ok
      All system wide checks as before
      Currently on file R4_1078
      13.1 MB/s to disk
      4.8 million items per second

15:35 Analysis of R4_1128 - attachment 10
Attachment 1: Screenshot_from_2022-05-15_08-38-04.png
Screenshot_from_2022-05-15_08-38-04.png
Attachment 2: Screenshot_from_2022-05-15_08-38-34.png
Screenshot_from_2022-05-15_08-38-34.png
Attachment 3: Screenshot_from_2022-05-15_08-39-12.png
Screenshot_from_2022-05-15_08-39-12.png
Attachment 4: Screenshot_from_2022-05-15_11-00-32.png
Screenshot_from_2022-05-15_11-00-32.png
Attachment 5: Screenshot_from_2022-05-15_11-00-50.png
Screenshot_from_2022-05-15_11-00-50.png
Attachment 6: Screenshot_from_2022-05-15_11-01-02.png
Screenshot_from_2022-05-15_11-01-02.png
Attachment 7: Screenshot_from_2022-05-15_13-13-50.png
Screenshot_from_2022-05-15_13-13-50.png
Attachment 8: Screenshot_from_2022-05-15_13-14-12.png
Screenshot_from_2022-05-15_13-14-12.png
Attachment 9: Screenshot_from_2022-05-15_13-14-31.png
Screenshot_from_2022-05-15_13-14-31.png
Attachment 10: R4_1128
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259195371 ( 1637875.8 Hz)
        Other data format:    2724629 (   17217.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2455 (      15.5 Hz)
                               RESUME:       2456 (      15.5 Hz)
                              SYNC100:      32590 (     205.9 Hz)
                              WR48-63:      32590 (     205.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2654538 (   16774.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      38026 (     240.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      158.251 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                4.735                0.000
  2                0.000                0.000
  3               16.674                0.000
  4                0.000                0.000
  5               77.034                0.000
  6                0.000                0.000
  7               33.677                0.000
  8                0.000                0.000
  9                0.000                0.000
 10                0.000               12.872
 11               17.247                0.000
 12                0.000                0.000
 13                0.000                8.200
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7464928       2802          0          0          0          0        927        927          0        948          0        661
  1   38695355      10162          0          0        254        254       4827       4827          0          0          0      32843
  2    6890959    1397581          0          0          0          0       1028       1028          0    1395525          0        633
  3   41303989    1261446          0          0        398        398       5227       5227          0    1250196          0        331
  4    4066993       1070          0          0          0          0        535        535          0          0          0        242
  5   37291979      10883          0          0        737        738       4704       4704          0          0          0        222
  6   12390385       7719          0          0          0          0       1626       1626          0       4467          0        394
  7   42901955      15088          0          0        626        626       5217       5217          0       3402          0        167
  8   10872099       2752          0          0          0          0       1376       1376          0          0          0        592
  9    4061050       1014          0          0          0          0        507        507          0          0          0        432
 10    2113468        548          0          0          0          0        274        274          0          0          0        291
 11   41561869      11172          0          0        440        440       5146       5146          0          0          0        468
 12    6798748       1722          0          0          0          0        861        861          0          0          0        436
 13    2781594        670          0          0          0          0        335        335          0          0          0        314
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.348s (  959.588 blocks/s,  59.974 Mb/s)
  455   Sat May 14 23:02:33 2022 TDSunday 15 May 00:00-08:00
Pb beam c. 1e+9/spill

00.02 zero all histograms
      system wide checks - baseline counters

00.08 ASIC check control - all FEE64s, all ASICs

Attachment 1 - grafana DSSSD bias, leakage current & temp - OK

Attachments 2 & 3 - ucesb - AIDA DSSSD #1 c. 10-20Hz peak rate on spill, c. 0Hz off spill

Attachment 4 - DSSSD bias & Leakage current - OK

Attachment 5 - FEE64 temps OK

Attachment 6-13 - adc, pause, resume & correlation scaler data items, push, flush, aida01, aida06

Attachments 14-18 - system wide checks - all OK *except* aida09 clock fail status 6

Attachments 19-25 - iptraf, TapeServer, NewMerger, NewMerger stats



00.44 file S450/R4_792

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator aida02 & aida04 0x16, aida06 & aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

00.51 disk space OK - c. 3' per data file

[npg@aidas-gsi S450]$ ls -l /
total 40
lrwxrwxrwx.   1 root root     7 Oct 18  2021 bin -> usr/bin
dr-xr-xr-x.   5 root root  4096 May 12 18:31 boot
drwxr-xr-x.  23 root root  4100 May 12 18:31 dev
drwxr-xr-x. 145 root root  8192 May 12 18:31 etc
drwxr-xr-x.   4 root root    56 Jan 28 14:56 home
lrwxrwxrwx.   1 root root     7 Oct 18  2021 lib -> usr/lib
lrwxrwxrwx.   1 root root     9 Oct 18  2021 lib64 -> usr/lib64
drwxr-xr-x.   5 root root   113 Apr 28 11:58 media
lrwxrwxrwx.   1 root root    45 Oct 18  2021 MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119
drwxr-xr-x.   2 root root     6 Apr 11  2018 mnt
drwxr-xr-x.   3 root root    16 Oct 18  2021 opt
dr-xr-xr-x. 464 root root     0 May 12 18:30 proc
dr-xr-x---.  10 root root  4096 May 13 11:47 root
drwxr-xr-x.  44 root root  1360 May 14 06:34 run
lrwxrwxrwx.   1 root root     8 Oct 18  2021 sbin -> usr/sbin
drwxr-xr-x.   2 root root     6 Apr 11  2018 srv
dr-xr-xr-x.  13 root root     0 May 12 18:31 sys
lrwxrwxrwx.   1 root root    28 Apr 28 14:19 TapeData -> /media/SecondDrive/TapeData/
drwxrwxrwt.  40 root root 12288 May 15 00:49 tmp
drwxr-xr-x.  13 root root   155 Oct 18  2021 usr
drwxr-xr-x.  21 root root  4096 Oct 18  2021 var
[npg@aidas-gsi S450]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G  373M  7.4G   5% /dev/shm
tmpfs                    7.8G   19M  7.7G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   16G   35G  31% /
/dev/sda2               1014M  226M  789M  23% /boot
/dev/sda1                200M   12M  189M   6% /boot/efi
/dev/sde1                7.2T  3.1T  3.8T  46% /media/SecondDrive
/dev/mapper/centos-home  407G   91G  316G  23% /home
tmpfs                    1.6G   52K  1.6G   1% /run/user/1000
/dev/sdd1                7.2T  6.5T  310G  96% /run/media/npg/ThirdDrive

Attachment 26 - analysis data file R4_792

Attachments 27-28 - per FEE64 rate & stat spectra - shows distro HEC events

Attachments 29-30 - per FEE64 1.8.L spectra 
 pulser peak widths aida01 134 ch FWHM, aida04 393 ch FWHM

Attachments 31-32 - per FEE64 1.8.H spectra

Attachments 33-34 - aida02 & aida04 1.*.H spectra

04.13

Attachment 35 - DSSSD bias & Leakage current - OK

Attachment 36 - FEE64 temps OK

Attachment 37 - adc data item stats

Attachments 38 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6


07.05

Attachment 39 - DSSSD bias & Leakage current - OK

Attachment 40 - FEE64 temps OK

Attachment 41 - adc data item stats

Attachments 42 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

Attachments 43-48 - aida02 & aida04 & aida06 & aida08 & 1.*.H spectra
Attachment 1: Screenshot_from_2022-05-15_00-03-33.png
Screenshot_from_2022-05-15_00-03-33.png
Attachment 2: Screenshot_from_2022-05-15_00-03-58.png
Screenshot_from_2022-05-15_00-03-58.png
Attachment 3: Screenshot_from_2022-05-15_00-04-18.png
Screenshot_from_2022-05-15_00-04-18.png
Attachment 4: Screenshot_from_2022-05-15_00-04-39.png
Screenshot_from_2022-05-15_00-04-39.png
Attachment 5: Screenshot_from_2022-05-15_00-05-10.png
Screenshot_from_2022-05-15_00-05-10.png
Attachment 6: Screenshot_from_2022-05-15_00-07-25.png
Screenshot_from_2022-05-15_00-07-25.png
Attachment 7: Screenshot_from_2022-05-15_00-16-59.png
Screenshot_from_2022-05-15_00-16-59.png
Attachment 8: Screenshot_from_2022-05-15_00-17-52.png
Screenshot_from_2022-05-15_00-17-52.png
Attachment 9: Screenshot_from_2022-05-15_00-18-55.png
Screenshot_from_2022-05-15_00-18-55.png
Attachment 10: Screenshot_from_2022-05-15_00-19-35.png
Screenshot_from_2022-05-15_00-19-35.png
Attachment 11: Screenshot_from_2022-05-15_00-20-14.png
Screenshot_from_2022-05-15_00-20-14.png
Attachment 12: Screenshot_from_2022-05-15_00-20-57.png
Screenshot_from_2022-05-15_00-20-57.png
Attachment 13: Screenshot_from_2022-05-15_00-21-32.png
Screenshot_from_2022-05-15_00-21-32.png
Attachment 14: Screenshot_from_2022-05-15_00-26-13.png
Screenshot_from_2022-05-15_00-26-13.png
Attachment 15: Screenshot_from_2022-05-15_00-26-49.png
Screenshot_from_2022-05-15_00-26-49.png
Attachment 16: Screenshot_from_2022-05-15_00-27-19.png
Screenshot_from_2022-05-15_00-27-19.png
Attachment 17: Screenshot_from_2022-05-15_00-28-00.png
Screenshot_from_2022-05-15_00-28-00.png
Attachment 18: Screenshot_from_2022-05-15_00-28-45.png
Screenshot_from_2022-05-15_00-28-45.png
Attachment 19: Screenshot_from_2022-05-15_00-33-46.png
Screenshot_from_2022-05-15_00-33-46.png
Attachment 20: Screenshot_from_2022-05-15_00-35-33.png
Screenshot_from_2022-05-15_00-35-33.png
Attachment 21: Screenshot_from_2022-05-15_00-36-00.png
Screenshot_from_2022-05-15_00-36-00.png
Attachment 22: Screenshot_from_2022-05-15_00-36-26.png
Screenshot_from_2022-05-15_00-36-26.png
Attachment 23: Screenshot_from_2022-05-15_00-36-58.png
Screenshot_from_2022-05-15_00-36-58.png
Attachment 24: Screenshot_from_2022-05-15_00-37-25.png
Screenshot_from_2022-05-15_00-37-25.png
Attachment 25: Screenshot_from_2022-05-15_00-37-46.png
Screenshot_from_2022-05-15_00-37-46.png
Attachment 26: R4_792
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  250102849 ( 1597022.2 Hz)
        Other data format:   11817151 (   75458.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2692 (      17.2 Hz)
                               RESUME:       2692 (      17.2 Hz)
                              SYNC100:      32584 (     208.1 Hz)
                              WR48-63:      32584 (     208.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   11746599 (   75007.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      48170 (     307.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      156.606 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                6.802                0.000
  2                0.000                0.000
  3               28.710                0.000
  4                0.000                0.000
  5               76.548                0.000
  6                0.000                0.000
  7               36.670                0.000
  8                0.002                0.000
  9                0.000                0.000
 10                0.000               14.137
 11               20.418                0.000
 12                0.000                0.000
 13                0.000                3.327
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7219269       2805          0          0          0          0        933        933          0        939          0        589
  1   37816150      10006          0          0        269        269       4734       4734          0          0          0      43094
  2    6973157    6465448          0          0          0          0       1676       1676          0    6462096          0        570
  3   37723722    5287878          0          0        579        580       5326       5326          0    5276067          0        352
  4    3982759       1044          0          0          0          0        522        522          0          0          0        233
  5   36626106      10369          0          0        747        746       4438       4438          0          0          0        211
  6   11915365       7321          0          0          0          0       1544       1544          0       4233          0        403
  7   42135728      15066          0          0        653        653       5248       5248          0       3264          0        174
  8   10616770       2532          0          0          2          2       1264       1264          0          0          0        570
  9    3975235        966          0          0          0          0        483        483          0          0          0        497
 10    2072466        538          0          0          0          0        269        269          0          0          0        289
 11   40106886      10836          0          0        442        442       4976       4976          0          0          0        472
 12    6251481       1636          0          0          0          0        818        818          0          0          0        413
 13    2687755        706          0          0          0          0        353        353          0          0          0        303
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.205s (  883.854 blocks/s,  55.241 Mb/s)
Attachment 27: Screenshot_from_2022-05-15_00-56-27.png
Screenshot_from_2022-05-15_00-56-27.png
Attachment 28: Screenshot_from_2022-05-15_00-57-44.png
Screenshot_from_2022-05-15_00-57-44.png
Attachment 29: Screenshot_from_2022-05-15_00-59-12.png
Screenshot_from_2022-05-15_00-59-12.png
Attachment 30: Screenshot_from_2022-05-15_01-02-36.png
Screenshot_from_2022-05-15_01-02-36.png
Attachment 31: Screenshot_from_2022-05-15_01-04-44.png
Screenshot_from_2022-05-15_01-04-44.png
Attachment 32: Screenshot_from_2022-05-15_01-05-49.png
Screenshot_from_2022-05-15_01-05-49.png
Attachment 33: Screenshot_from_2022-05-15_01-13-22.png
Screenshot_from_2022-05-15_01-13-22.png
Attachment 34: Screenshot_from_2022-05-15_01-15-01.png
Screenshot_from_2022-05-15_01-15-01.png
Attachment 35: Screenshot_from_2022-05-15_04-13-03.png
Screenshot_from_2022-05-15_04-13-03.png
Attachment 36: Screenshot_from_2022-05-15_04-13-31.png
Screenshot_from_2022-05-15_04-13-31.png
Attachment 37: Screenshot_from_2022-05-15_04-13-52.png
Screenshot_from_2022-05-15_04-13-52.png
Attachment 38: Screenshot_from_2022-05-15_04-14-31.png
Screenshot_from_2022-05-15_04-14-31.png
Attachment 39: Screenshot_from_2022-05-15_07-05-19.png
Screenshot_from_2022-05-15_07-05-19.png
Attachment 40: Screenshot_from_2022-05-15_07-05-50.png
Screenshot_from_2022-05-15_07-05-50.png
Attachment 41: Screenshot_from_2022-05-15_07-06-15.png
Screenshot_from_2022-05-15_07-06-15.png
Attachment 42: Screenshot_from_2022-05-15_07-06-42.png
Screenshot_from_2022-05-15_07-06-42.png
Attachment 43: Screenshot_from_2022-05-15_07-06-42.png
Screenshot_from_2022-05-15_07-06-42.png
Attachment 44: Screenshot_from_2022-05-15_07-12-37.png
Screenshot_from_2022-05-15_07-12-37.png
Attachment 45: Screenshot_from_2022-05-15_07-14-49.png
Screenshot_from_2022-05-15_07-14-49.png
Attachment 46: Screenshot_from_2022-05-15_07-16-18.png
Screenshot_from_2022-05-15_07-16-18.png
Attachment 47: Screenshot_from_2022-05-15_07-20-47.png
Screenshot_from_2022-05-15_07-20-47.png
Attachment 48: Screenshot_from_2022-05-15_07-22-02.png
Screenshot_from_2022-05-15_07-22-02.png
  454   Sat May 14 21:34:11 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Attachment 1: Rates_2022-05-14_22-32-44.png
Rates_2022-05-14_22-32-44.png
Attachment 2: Temp_2022-05-14_22-31-37.png
Temp_2022-05-14_22-31-37.png
Attachment 3: Leakg_2022-05-14_22-30-28.png
Leakg_2022-05-14_22-30-28.png
  453   Sat May 14 19:39:13 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Attachment 1: Stat_2022-05-14_20-36-15.png
Stat_2022-05-14_20-36-15.png
Attachment 2: Temp_2022-05-14_20-35-19.png
Temp_2022-05-14_20-35-19.png
Attachment 3: Leakage_2022-05-14_20-33-32.png
Leakage_2022-05-14_20-33-32.png
  452   Sat May 14 17:34:40 2022 BA, AASaturday 14 May

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 13, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 14, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Attachment 1: Stat_2022-05-14_18-33-16.png
Stat_2022-05-14_18-33-16.png
Attachment 2: Temp_2022-05-14_18-32-07.png
Temp_2022-05-14_18-32-07.png
Attachment 3: Leakage_2022-05-14_18-30-52.png
Leakage_2022-05-14_18-30-52.png
  451   Sat May 14 15:35:24 2022 BA, AASaturday 14 May
Attachment 1: Stais_from_2022-05-14_16-30-53.png
Stais_from_2022-05-14_16-30-53.png
Attachment 2: Temp_2022-05-14_16-32-42.png
Temp_2022-05-14_16-32-42.png
Attachment 3: Leakage_2022-05-14_16-33-45.png
Leakage_2022-05-14_16-33-45.png
  450   Sat May 14 13:01:58 2022 MS, OHSaturday 14 May

14:00

Attachment 1: Screenshot_2022-05-14_at_14-02-31_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_14-02-31_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_14-02-52_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_14-02-52_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_14-03-54.png
Screenshot_from_2022-05-14_14-03-54.png
  449   Sat May 14 10:59:52 2022 MS, OHSaturday 14 May

12:00

Attachment 1: Screenshot_2022-05-14_at_12-00-16_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_12-00-16_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_12-00-44_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_12-00-44_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_12-01-54.png
Screenshot_from_2022-05-14_12-01-54.png
  448   Sat May 14 09:00:21 2022 MS, OHSaturday 14 May

10:00

Attachment 1: Screenshot_2022-05-14_at_10-01-07_Statistics_aidas-gsi.png
Screenshot_2022-05-14_at_10-01-07_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-14_at_10-01-50_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-14_at_10-01-50_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-14_10-02-51.png
Screenshot_from_2022-05-14_10-02-51.png
  447   Sat May 14 06:57:21 2022 MS, OHSaturday 14 May
08:00 Took over the shift from Philippos
      In system wide checks aida09 fails ASIC clock check but it is bit 6 which is ok
  446   Sat May 14 03:14:00 2022 PPShift Checks

4:13

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

 

Screenshot 1: statistics

Screenshot 2: temperatures

Screenshot 3: scalers

Screenshot 4: Bias and leakage current

 

6:36

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR


Screenshot 5: statistics

Screenshot 6: temperatures

Screenshot 7: scalers

Screenshot 8: Bias and leakage current

 

Attachment 1: stats-220514-0428.png
stats-220514-0428.png
Attachment 2: temps-220514-0417.png
temps-220514-0417.png
Attachment 3: scalers-220514-0426.png
scalers-220514-0426.png
Attachment 4: I-V-220514-0431.png
I-V-220514-0431.png
Attachment 5: stats-220514-0639.png
stats-220514-0639.png
Attachment 6: temps-220514-0639.png
temps-220514-0639.png
Attachment 7: scalers-220514-0641.png
scalers-220514-0641.png
Attachment 8: I-V-220514-0640.png
I-V-220514-0640.png
  445   Sat May 14 01:52:03 2022 ML2h-Shift Checks

AIDA stats ok

Leakage current ok

Temperatires ok

grafana ok

ucesb rates dropped a bit in all scalers so beam intensity must have dropped a bit.

System wide check done and same results as earlier:

aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Attachment 1: Stats_Screenshot_from_2022-05-14_02-48-14.png
Stats_Screenshot_from_2022-05-14_02-48-14.png
Attachment 2: Temp_Screenshot_from_2022-05-14_02-47-31.png
Temp_Screenshot_from_2022-05-14_02-47-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-14_02-46-42.png
HV_LC_Screenshot_from_2022-05-14_02-46-42.png
Attachment 4: ucesb_rates_Screenshot_from_2022-05-14_02-50-36.png
ucesb_rates_Screenshot_from_2022-05-14_02-50-36.png
Attachment 5: Grafana_Screenshot_from_2022-05-14_03-50-00.png
Grafana_Screenshot_from_2022-05-14_03-50-00.png
  444   Fri May 13 23:37:59 2022 ML2h-Shift Checks
Took over remotely from CB  about an hour ago

At 0h40 (CET):
      Stats still ok - attach 1
      Leakage still OK - Attach 2
      Temps - attach 3
      Grafana OK
      UCESB rates ok - attach 5
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR
Attachment 1: Stats_Screenshot_from_2022-05-14_00-40-56.png
Stats_Screenshot_from_2022-05-14_00-40-56.png
Attachment 2: HV_LC_Screenshot_from_2022-05-14_00-39-51.png
HV_LC_Screenshot_from_2022-05-14_00-39-51.png
Attachment 3: temp_Screenshot_from_2022-05-14_00-41-44.png
temp_Screenshot_from_2022-05-14_00-41-44.png
Attachment 4: UCESB_Screenshot_from_2022-05-14_01-00-00.png
UCESB_Screenshot_from_2022-05-14_01-00-00.png
  443   Fri May 13 15:00:23 2022 CB13 March - 16:00-24:00 shift
16:00 Took over remotely from OH, TD who remain in GSI

      Stats OK - attach 1
      Stats & Leakage OK - attach 2
      Temps OK
      Grafana OK - attach 3
      
      System-wide checks: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

17:29 All good, but no beam.
17:30 Beam is back

17:58 No beam again
18:10 Beam is back. Stats OK.

      Ucesb OK - attach 4
      Stats & Leakage currents ok - attach 5
      Temps OK - attach 6
      Grafana OK - attach 7

      System-wide checks as before. 

19:36 No beam again.
20:37 Beam is back

      Stats OK - attach 8
      Leakage OK
      Temps OK
      Grafana OK - attach 9
      Ucesb OK - attach 10

      System-wide checks as before

21:10 Lost anydesk connection to DAQ. Local shifters report AIDA still transferring data.
21:15 Local shifters report anydesk connection window still open. Unclear why DAQ reported offline in anydesk.
21:20 Connection restored itself. Hopefully it was a one-off.

22:51 Stats & Leakage current OK - attach 11
      Temps OK
      Grafana OK - attach 12
      Ucesb OK - attach 13

      System-wide checks as before
Attachment 1: 1305_1600_ucesb.png
1305_1600_ucesb.png
Attachment 2: 1305_1600_StatsTemps.png
1305_1600_StatsTemps.png
Attachment 3: 1305_1600_Grafana.png
1305_1600_Grafana.png
Attachment 4: 1305_1800_StatsLeakage.png
1305_1800_StatsLeakage.png
Attachment 5: 1305_2030_Stats.png
1305_2030_Stats.png
Attachment 6: 1305_2030_ucesb.png
1305_2030_ucesb.png
Attachment 7: 1305_2030_Grafana.png
1305_2030_Grafana.png
Attachment 8: 1305_2300_StatsLeakage.png
1305_2300_StatsLeakage.png
Attachment 9: 1305_2300_ucesb.png
1305_2300_ucesb.png
Attachment 10: 1305_2300_Grafana.png
1305_2300_Grafana.png
  442   Fri May 13 07:36:18 2022 OH, TDFriday 13 May
08:36 DAQ Still running smoothly
      Currently on run R1_328
      At current data rates we have just over 4 days of space left on the current disk
      Analysis of R1_327 - attachment 1

14.15 all histograms zero'd
      baseline system wide checks counters
 
      DAQ continues file S450/R4_33

      netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
      netint flushenable 60 - not changed 

      ASIC settings 
       slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      frequency 2Hz
      tau_d 1ms (tail pulse)
      + polarity

Attachments 2 & 3 - ucesb
      Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.

Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK

Attachments 6-8 - NewMerger stats blocks 0, 1, & 6

Attachment 9 - NewMerger link stats

Attachments 10-11 - TapeServer/NewMerger

Attachment 12 - iptraf aida-gsi network interfaces

Attachments 13-14 - per FEE64 1.8.L spectra
 pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM

Attachments 15-18 - per FEE64 rates and stat spectra

Attachments 19-22 - system wide checks 
 aida09 clock status 6, WR decoder status errors

Attachments 23-26 - adc, pause, resume correlation sclaer data item stats

Attachment 27 - FEE64 temps OK

Attachment 28 - DSSSSD bias and leakage currents OK

14.30 check ASIC load

14.47 attachment 29 - analysis S450/R4_38

15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate)
Attachment 1: 220513_0839_R1_327_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259232212 ( 1719602.4 Hz)
        Other data format:    2687788 (   17829.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        986 (       6.5 Hz)
                               RESUME:        986 (       6.5 Hz)
                              SYNC100:      32562 (     216.0 Hz)
                              WR48-63:      32562 (     216.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2620692 (   17384.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     130683 (     866.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      150.751 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.085                0.000
  1                9.978                0.000
  2                0.059                0.000
  3               21.640                0.000
  4                0.158                0.000
  5               66.897                0.000
  6                1.778                0.000
  7               36.972                0.000
  8                0.172                0.000
  9                0.018                0.000
 10                0.000                7.315
 11               28.135                0.000
 12                0.007                0.000
 13                0.000                3.034
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7933827       2887          0          0          5          5        987        987          0        903          0       1141
  1   37526918       9508          0          0         50         50       4704       4704          0          0          0     122502
  2    8368319    1400718          0          0          2          2       1222       1222          0    1398270          0        647
  3   39185673    1207805          0          0        156        156       5116       5116          0    1197261          0        703
  4    4609566       1178          0          0          5          5        584        584          0          0          0       1170
  5   40049357      10540          0          0        309        309       4961       4961          0          0          0        297
  6   13162553      17502          0          0         13         13       1685       1685          0      14106          0        302
  7   41199857      20850          0          0        248        248       5101       5101          0      10152          0        241
  8   12021437       2864          0          0          5          5       1427       1427          0          0          0        467
  9    4748197       1188          0          0          2          2        592        592          0          0          0       1165
 10    3089949        744          0          0          0          0        372        372          0          0          0        247
 11   38258323       9856          0          0        190        190       4738       4738          0          0          0       1052
 12    6307578       1518          0          0          1          1        758        758          0          0          0        311
 13    2770658        630          0          0          0          0        315        315          0          0          0        438
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.600s (  952.392 blocks/s,  59.525 Mb/s)
Attachment 2: Screenshot_from_2022-05-13_14-28-00.png
Screenshot_from_2022-05-13_14-28-00.png
Attachment 3: Screenshot_from_2022-05-13_14-27-51.png
Screenshot_from_2022-05-13_14-27-51.png
Attachment 4: Screenshot_from_2022-05-13_14-27-31.png
Screenshot_from_2022-05-13_14-27-31.png
Attachment 5: Screenshot_from_2022-05-13_14-27-17.png
Screenshot_from_2022-05-13_14-27-17.png
Attachment 6: Screenshot_from_2022-05-13_14-26-32.png
Screenshot_from_2022-05-13_14-26-32.png
Attachment 7: Screenshot_from_2022-05-13_14-26-26.png
Screenshot_from_2022-05-13_14-26-26.png
Attachment 8: Screenshot_from_2022-05-13_14-26-21.png
Screenshot_from_2022-05-13_14-26-21.png
Attachment 9: Screenshot_from_2022-05-13_14-26-08.png
Screenshot_from_2022-05-13_14-26-08.png
Attachment 10: Screenshot_from_2022-05-13_14-25-46.png
Screenshot_from_2022-05-13_14-25-46.png
Attachment 11: Screenshot_from_2022-05-13_14-25-38.png
Screenshot_from_2022-05-13_14-25-38.png
Attachment 12: Screenshot_from_2022-05-13_14-25-22.png
Screenshot_from_2022-05-13_14-25-22.png
Attachment 13: Screenshot_from_2022-05-13_14-24-07.png
Screenshot_from_2022-05-13_14-24-07.png
Attachment 14: Screenshot_from_2022-05-13_14-22-44.png
Screenshot_from_2022-05-13_14-22-44.png
Attachment 15: Screenshot_from_2022-05-13_14-21-16.png
Screenshot_from_2022-05-13_14-21-16.png
Attachment 16: Screenshot_from_2022-05-13_14-21-04.png
Screenshot_from_2022-05-13_14-21-04.png
Attachment 17: Screenshot_from_2022-05-13_14-20-34.png
Screenshot_from_2022-05-13_14-20-34.png
Attachment 18: Screenshot_from_2022-05-13_14-20-14.png
Screenshot_from_2022-05-13_14-20-14.png
Attachment 19: Screenshot_from_2022-05-13_14-19-10.png
Screenshot_from_2022-05-13_14-19-10.png
Attachment 20: Screenshot_from_2022-05-13_14-18-40.png
Screenshot_from_2022-05-13_14-18-40.png
Attachment 21: Screenshot_from_2022-05-13_14-18-31.png
Screenshot_from_2022-05-13_14-18-31.png
Attachment 22: Screenshot_from_2022-05-13_14-18-14.png
Screenshot_from_2022-05-13_14-18-14.png
Attachment 23: Screenshot_from_2022-05-13_14-17-38.png
Screenshot_from_2022-05-13_14-17-38.png
Attachment 24: Screenshot_from_2022-05-13_14-17-23.png
Screenshot_from_2022-05-13_14-17-23.png
Attachment 25: Screenshot_from_2022-05-13_14-17-09.png
Screenshot_from_2022-05-13_14-17-09.png
Attachment 26: Screenshot_from_2022-05-13_14-16-56.png
Screenshot_from_2022-05-13_14-16-56.png
Attachment 27: Screenshot_from_2022-05-13_14-16-30.png
Screenshot_from_2022-05-13_14-16-30.png
Attachment 28: Screenshot_from_2022-05-13_14-16-11.png
Screenshot_from_2022-05-13_14-16-11.png
Attachment 29: R4_38.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259102731 ( 1572653.5 Hz)
        Other data format:    2817271 (   17099.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2412 (      14.6 Hz)
                               RESUME:       2414 (      14.7 Hz)
                              SYNC100:      32614 (     198.0 Hz)
                              WR48-63:      32614 (     198.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2747217 (   16674.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      30594 (     185.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      164.755 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                5.297                0.000
  2                0.000                0.000
  3               18.287                0.000
  4                0.000                0.000
  5               77.894                0.000
  6                0.000                0.000
  7               25.402                0.000
  8                0.001                0.000
  9                0.000                0.000
 10                0.000                2.705
 11               15.617                0.000
 12                0.000                0.000
 13                0.000               29.550
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6844719       2775          0          0          0          0        894        894          0        987          0        375
  1   39920437      10378          0          0        277        277       4912       4912          0          0          0      27546
  2    7063505    1453891          0          0          0          0       1085       1085          0    1451721          0        328
  3   42633132    1301928          0          0        410        410       5590       5590          0    1289928          0        251
  4    3625899        900          0          0          0          0        450        450          0          0          0        125
  5   38867835      10853          0          0        768        769       4658       4658          0          0          0        129
  6    7590830       4334          0          0          0          0        934        934          0       2466          0        227
  7   44498336      14120          0          0        564        565       5438       5438          0       2115          0        113
  8   10949373       2716          0          0          1          1       1357       1357          0          0          0        345
  9    3887105        996          0          0          0          0        498        498          0          0          0        273
 10    1999345        532          0          0          0          0        266        266          0          0          0        160
 11   43113077      11754          0          0        392        392       5485       5485          0          0          0        288
 12    6030564       1596          0          0          0          0        798        798          0          0          0        252
 13    2078574        498          0          0          0          0        249        249          0          0          0        182
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.668s (  979.553 blocks/s,  61.222 Mb/s)
Attachment 30: Screenshot_from_2022-05-13_15-11-36.png
Screenshot_from_2022-05-13_15-11-36.png
Attachment 31: Screenshot_from_2022-05-13_15-11-29.png
Screenshot_from_2022-05-13_15-11-29.png
Attachment 32: Screenshot_from_2022-05-13_15-11-17.png
Screenshot_from_2022-05-13_15-11-17.png
Attachment 33: Screenshot_from_2022-05-13_15-10-55.png
Screenshot_from_2022-05-13_15-10-55.png
Attachment 34: Screenshot_from_2022-05-13_15-10-41.png
Screenshot_from_2022-05-13_15-10-41.png
Attachment 35: Screenshot_from_2022-05-13_15-10-30.png
Screenshot_from_2022-05-13_15-10-30.png
  441   Fri May 13 06:56:14 2022 OH, BA, MALow Rates

 

Quote:

The rate is very low in all aida, not sure .

This was because the statistics page had been changed to transfer buffers. The rates were ok

  440   Fri May 13 06:47:33 2022 ML2h-Shift Checks

It has been a quiet night. AIDA has been stable.

Stats appears stable appear

Temperature also stable

Leakage current slowly dropping still.

System wide checks done. Status quo here again.

Attachment 1: Stats_Screenshot_from_2022-05-13_07-43-54.png
Stats_Screenshot_from_2022-05-13_07-43-54.png
Attachment 2: Temp_Screenshot_from_2022-05-13_07-43-13.png
Temp_Screenshot_from_2022-05-13_07-43-13.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_07-42-38.png
HV_LC_Screenshot_from_2022-05-13_07-42-38.png
  439   Fri May 13 04:45:23 2022 ML2h-Shift Checks

AIDA is running ok. Note that, in the last hour or so, the rate of implant in DSSD1 is back to higher rate ~10 Hzto 15 Hz as at the beginning of this shift.

Stats seems ok

Temperatures stable

Leakage current: ok

A systwm wide check gives same result as for the fist check on this shift at ~1am

Attachment 1: Stats_Screenshot_from_2022-05-13_05-43-14.png
Stats_Screenshot_from_2022-05-13_05-43-14.png
Attachment 2: Temp_Screenshot_from_2022-05-13_05-42-10.png
Temp_Screenshot_from_2022-05-13_05-42-10.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_05-41-23.png
HV_LC_Screenshot_from_2022-05-13_05-41-23.png
  438   Fri May 13 02:56:36 2022 ML2h-Shift Checks

All seemnormal on AIDA side.

Stats as before

Temperatures stable

HV-LV ok

System wide check : all results as in the previous checks

Attachment 1: Stats_Screenshot_from_2022-05-13_03-43-34.png
Stats_Screenshot_from_2022-05-13_03-43-34.png
Attachment 2: Temp_Screenshot_from_2022-05-13_03-44-41.png
Temp_Screenshot_from_2022-05-13_03-44-41.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_03-41-28.png
HV_LC_Screenshot_from_2022-05-13_03-41-28.png
  437   Fri May 13 02:28:36 2022 MLbeam dropped

We see less counts in AIDA implant DSSD1 (~1Hz) than at the beginning of the night shift (~10Hz).

This seems to be beam related as the count rates in DESPEC & FRS Scalers seems to have dropped too.

  436   Fri May 13 00:50:16 2022 ML2h-Shift Checks

All seems ok  (apart for my virgin network connection usually quite reliable but not so tonight !?!).

Statistics seems ok.

Temperatures appears stable

Leakage current ok.

System Wide Checks:

Clock check: Passed= 14: - Failed=: 0

ADC calibration check:

FEE64 module aida01 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida13 failed
Calibration test result: Passed=10 ; Failed= 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module -> DID THAT BUT NO CHANGES

White rabbit decode status: Passed=14 ; Failed=0

FPGA timestamp check: Passed=14 ; Failed=0

Memory information: Page loading failed with the error message below:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
could not read "/Embedded/XilinxLinux/ppc_4xx/rfs/aida01/tmp/mybuddy.txt": no such file or directory while executing "file size $mybuddy " (procedure "do_collect_info" line 16) invoked from within "do_collect_info " ("COLLECTINFO" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"

Attachment 1: Stats_Screenshot_from_2022-05-13_01-45-56.png
Stats_Screenshot_from_2022-05-13_01-45-56.png
Attachment 2: Temp_Screenshot_from_2022-05-13_01-47-31.png
Temp_Screenshot_from_2022-05-13_01-47-31.png
Attachment 3: HV_LC_Screenshot_from_2022-05-13_01-48-21.png
HV_LC_Screenshot_from_2022-05-13_01-48-21.png
  435   Thu May 12 22:49:39 2022 BA, MALow Rates

The rate is very low in all aida, not sure .

Attachment 1: RatesLow2022-05-12_23-48-01.png
RatesLow2022-05-12_23-48-01.png
  434   Thu May 12 21:43:49 2022 BA, MARates

 

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

 

Attachment 1: Stati2022-05-12_22-47-26.png
Stati2022-05-12_22-47-26.png
Attachment 2: Temp2022-05-12_22-48-41.png
Temp2022-05-12_22-48-41.png
Attachment 3: LeakageCurrent2022-05-12_22-45-44.png
LeakageCurrent2022-05-12_22-45-44.png
  433   Thu May 12 16:30:47 2022 NH, OH, TDNotes for mounting a second switch
Update DHCPD as expected
Make sure new ethernet device is trusted in firewall
Make sure hosts is updated
make sure /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08/etc/hosts "myserver" points to correct ip address
make sure /MIDAS/linux-ppc_4xx/startup/aidaXX has netvar "TS_Server" pointing to correct ip address
Update the IP in /MIDAS/linux-ppc_4xx/startup/aida*
  432   Thu May 12 10:10:51 2022 TD, OHAIDA data throughput
Attachment 1 - top aida06

Attachment 2 - top aida01

Attachment 3 - top aida-gsi (AIDA server)

Attachment 4 - TapeServer

Attachment 5 - NewMerger stats block 6 (aida06)

Attachment 6 - NewMerger stats block 1 (aida01)

Attachment 7 - NewMerger stats block 0 (all FEE64s)

Attachment 8 - NewMerger options

Attachment 9 - NewMerger link stats

Attachment 10 - NewMerger

Attachments 11-14 - ADC, pause, resume & correlation scaler data item stats

Attacgment 15 - Network traffic
Attachment 1: Screenshot_from_2022-05-12_11-08-34.png
Screenshot_from_2022-05-12_11-08-34.png
Attachment 2: Screenshot_from_2022-05-12_11-07-44.png
Screenshot_from_2022-05-12_11-07-44.png
Attachment 3: Screenshot_from_2022-05-12_11-07-14.png
Screenshot_from_2022-05-12_11-07-14.png
Attachment 4: Screenshot_from_2022-05-12_11-05-53.png
Screenshot_from_2022-05-12_11-05-53.png
Attachment 5: Screenshot_from_2022-05-12_11-05-40.png
Screenshot_from_2022-05-12_11-05-40.png
Attachment 6: Screenshot_from_2022-05-12_11-05-18.png
Screenshot_from_2022-05-12_11-05-18.png
Attachment 7: Screenshot_from_2022-05-12_11-05-09.png
Screenshot_from_2022-05-12_11-05-09.png
Attachment 8: Screenshot_from_2022-05-12_11-04-46.png
Screenshot_from_2022-05-12_11-04-46.png
Attachment 9: Screenshot_from_2022-05-12_11-04-22.png
Screenshot_from_2022-05-12_11-04-22.png
Attachment 10: Screenshot_from_2022-05-12_11-03-49.png
Screenshot_from_2022-05-12_11-03-49.png
Attachment 11: Screenshot_from_2022-05-12_11-03-37.png
Screenshot_from_2022-05-12_11-03-37.png
Attachment 12: Screenshot_from_2022-05-12_11-03-28.png
Screenshot_from_2022-05-12_11-03-28.png
Attachment 13: Screenshot_from_2022-05-12_11-03-13.png
Screenshot_from_2022-05-12_11-03-13.png
Attachment 14: Screenshot_from_2022-05-12_11-02-58.png
Screenshot_from_2022-05-12_11-02-58.png
Attachment 15: Screenshot_from_2022-05-12_11-15-56.png
Screenshot_from_2022-05-12_11-15-56.png
  431   Thu May 12 07:09:43 2022 TDThursday 12 May 08:00-24:00
08.10 AIDA DAQ status
    
      14x FEE64s configured - aida05, aida14 power OFF - dhcpd.conf changed - aida15 -> aida05, aida16 -> aida14
2
      AD9222 OFF - waveforms disabed
      disc - all disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator 0xc *except* aida02 & aida04 0x16, aida06 & aida08 0x23

      BNC PB-5
      amplitude 1.000V
      attenuation x1
      tau_d 1ms (tail pulse)
      polarity +
      frequency 22Hz

08.14 all histograms zero'd
2
08.15 system wide check counter baseline

08.17 check ASIC control - all FEE64s all ASICs

Attachment 1 - grafana DSSSD bias & leakage current - OK

Attachment 2 - per FEE64 stat spectra

Attachments 3 & 4 - per FEE64 1.8.H spectra - few events c. 5GeV p+n strips

Attachments 5 & 6 - per FEE64 1.8.L spectra
 pulser peak width aida01 124ch FWHM, aida04 293ch FWHM

Attachment 7 - per FEE64 rate spectra

Attachments 8-12 - system wide checks - all OK

Attachments 13-16 - statistics - adc, pause, resume & correlation data

Attachment 17 - FEE64 temps OK

Attachment 18 - DSSSD bias & leakage current OK - start new section of googledoc spreadsheet to monitor long term trends

Attachments 19 & 20 - NewMerger & TapeServer - OK - no warning messages

08.44 DAQ continues - file NOTAPE/R18 - no storage mode

08.47 switch to storage mode to generate data file to evaluate FEE64 dead times etc

08.53 no storage mode enabled

Attachment 21 - analysis of file NOTAPE/R18_20

Attachment 22- analysis of file NOTAPE/R19_19
               AIDA12 slow comp up to 0xd improved deadtime drastically for FEE

10:54 Beam on detector. Take another run to test deadtime - attachment 23
      Deadtimes ranging from 10-40% in n+n FEEs

Attachment 23 & 24 aida06 aida02 1.*.H spectra


14:07 - Currently no beam so had acces. OH checked the cables on AIDA12 were seated correctly. No improvement in the rate for 12 but better for remaining p+n strips.
        Tested running with dataspy off for PCS but observed no change in the deadtimes (Other than a decrease because no beamtime)
Attachment 1: Screenshot_from_2022-05-12_08-25-44.png
Screenshot_from_2022-05-12_08-25-44.png
Attachment 2: Screenshot_from_2022-05-12_08-25-20.png
Screenshot_from_2022-05-12_08-25-20.png
Attachment 3: Screenshot_from_2022-05-12_08-24-01.png
Screenshot_from_2022-05-12_08-24-01.png
Attachment 4: Screenshot_from_2022-05-12_08-23-35.png
Screenshot_from_2022-05-12_08-23-35.png
Attachment 5: Screenshot_from_2022-05-12_08-22-24.png
Screenshot_from_2022-05-12_08-22-24.png
Attachment 6: Screenshot_from_2022-05-12_08-21-18.png
Screenshot_from_2022-05-12_08-21-18.png
Attachment 7: Screenshot_from_2022-05-12_08-19-15.png
Screenshot_from_2022-05-12_08-19-15.png
Attachment 8: Screenshot_from_2022-05-12_08-16-40.png
Screenshot_from_2022-05-12_08-16-40.png
Attachment 9: Screenshot_from_2022-05-12_08-16-30.png
Screenshot_from_2022-05-12_08-16-30.png
Attachment 10: Screenshot_from_2022-05-12_08-16-25.png
Screenshot_from_2022-05-12_08-16-25.png
Attachment 11: Screenshot_from_2022-05-12_08-16-17.png
Screenshot_from_2022-05-12_08-16-17.png
Attachment 12: Screenshot_from_2022-05-12_08-16-08.png
Screenshot_from_2022-05-12_08-16-08.png
Attachment 13: Screenshot_from_2022-05-12_08-10-02.png
Screenshot_from_2022-05-12_08-10-02.png
Attachment 14: Screenshot_from_2022-05-12_08-09-44.png
Screenshot_from_2022-05-12_08-09-44.png
Attachment 15: Screenshot_from_2022-05-12_08-09-20.png
Screenshot_from_2022-05-12_08-09-20.png
Attachment 16: Screenshot_from_2022-05-12_08-09-02.png
Screenshot_from_2022-05-12_08-09-02.png
Attachment 17: Screenshot_from_2022-05-12_08-08-39.png
Screenshot_from_2022-05-12_08-08-39.png
Attachment 18: Screenshot_from_2022-05-12_08-08-26.png
Screenshot_from_2022-05-12_08-08-26.png
Attachment 19: Screenshot_from_2022-05-12_08-39-14.png
Screenshot_from_2022-05-12_08-39-14.png
Attachment 20: Screenshot_from_2022-05-12_08-39-05.png
Screenshot_from_2022-05-12_08-39-05.png
Attachment 21: R18_20
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259331519 ( 1693112.6 Hz)
        Other data format:    2588481 (   16899.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1016 (       6.6 Hz)
                               RESUME:       1017 (       6.6 Hz)
                              SYNC100:      32570 (     212.6 Hz)
                              WR48-63:      32570 (     212.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2521308 (   16461.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      153.169 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.171                0.000
  1                9.417                0.000
  2                0.061                0.000
  3               24.348                0.000
  4                0.008                0.000
  5               72.336                0.000
  6                0.231                0.000
  7               25.570                0.000
  8                0.125                0.000
  9                0.007                0.000
 10                0.000                7.342
 11               41.574                0.000
 12                0.027                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7431923      12514          0          0          8          8        933        933          0      10632          0          0
  1   37380104       9380          0          0         51         51       4639       4639          0          0          0          0
  2    7217279    1365495          0          0          3          3       1056       1056          0    1363377          0          0
  3   38015689    1157497          0          0        179        179       4920       4920          0    1147299          0          0
  4    4440445       1078          0          0          2          2        537        537          0          0          0          0
  5   40380917      10697          0          0        319        320       5029       5029          0          0          0          0
  6   14913071       3788          0          0          5          5       1889       1889          0          0          0          0
  7   41232478      10490          0          0        181        181       5064       5064          0          0          0          0
  8   11952709       2932          0          0          4          4       1462       1462          0          0          0          0
  9    4910914       1250          0          0          1          1        624        624          0          0          0          0
 10    2469929        580          0          0          0          0        290        290          0          0          0          0
 11   39767127      10486          0          0        257        257       4986       4986          0          0          0          0
 12    5557988       1406          0          0          6          6        697        697          0          0          0          0
 13    3660946        888          0          0          0          0        444        444          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   36.578s (  874.840 blocks/s,  54.677 Mb/s)
Attachment 22: R19_19.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      17139
          ADC data format:  138519076 ( 1563611.9 Hz)
        Other data format:    1763639 (   19908.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        358 (       4.0 Hz)
                               RESUME:        358 (       4.0 Hz)
                              SYNC100:      17469 (     197.2 Hz)
                              WR48-63:      17469 (     197.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1727985 (   19505.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       88.589 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.084                0.000
  1                0.528                0.000
  2                0.008                0.000
  3               10.973                0.000
  4                0.000                0.000
  5               30.421                0.000
  6                0.000                0.000
  7                7.252                0.000
  8                0.055                0.000
  9                0.000                0.000
 10                0.000               12.541
 11                3.332                0.000
 12                0.009                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    4390551       7288          0          0          4          4        565        565          0       6150          0          0
  1   21021919       5186          0          0          9          9       2584       2584          0          0          0          0
  2    3174721     918030          0          0          1          1        518        518          0     916992          0          0
  3   22097261     810837          0          0         77         77       2920       2920          0     804843          0          0
  4    2596723        620          0          0          0          0        310        310          0          0          0          0
  5   23543560       6240          0          0        174        174       2946       2946          0          0          0          0
  6    4211356       1020          0          0          0          0        510        510          0          0          0          0
  7   24080430       6296          0          0         63         63       3085       3085          0          0          0          0
  8    4823696       1068          0          0          3          3        531        531          0          0          0          0
  9    1770702        408          0          0          0          0        204        204          0          0          0          0
 10     933213        222          0          0          0          0        111        111          0          0          0          0
 11   21419475       5280          0          0         21         21       2619       2619          0          0          0          0
 12    2929744        698          0          0          6          6        343        343          0          0          0          0
 13    1525725        446          0          0          0          0        223        223          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   17.484s (  980.247 blocks/s,  61.265 Mb/s)
Attachment 23: R20_20.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       4909
          ADC data format:   39765494 ( 1644562.3 Hz)
        Other data format:     414671 (   17149.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        151 (       6.2 Hz)
                               RESUME:        152 (       6.3 Hz)
                              SYNC100:       4999 (     206.7 Hz)
                              WR48-63:       4999 (     206.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:     404370 (   16723.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      23252 (     961.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       24.180 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                1.954                0.000
  2                0.000                0.000
  3                4.569                0.000
  4                0.000                0.000
  5               10.824                0.000
  6                0.000                0.000
  7                5.869                0.000
  8                0.000                0.000
  9                0.008                0.000
 10                0.000                0.000
 11                2.942                0.000
 12                0.007                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    1157062       1976          0          0          0          0        148        148          0       1680          0        159
  1    5834981       1480          0          0         12         12        728        728          0          0          0      22412
  2    1125294     220409          0          0          0          0        178        178          0     220053          0         83
  3    5924210     180469          0          0         32         32        741        741          0     178923          0         86
  4     669291        180          0          0          0          0         90         90          0          0          0         10
  5    6069826       1577          0          0         47         48        741        741          0          0          0         20
  6    1749760       2346          0          0          0          0        222        222          0       1902          0         32
  7    6604292       3590          0          0         38         38        851        851          0       1812          0         32
  8    1794890        416          0          0          0          0        208        208          0          0          0         42
  9     736012        164          0          0          1          1         81         81          0          0          0        124
 10     410664         98          0          0          0          0         49         49          0          0          0         15
 11    6315369       1576          0          0         19         19        769        769          0          0          0        153
 12     852659        248          0          0          2          2        122        122          0          0          0         23
 13     521184        142          0          0          0          0         71         71          0          0          0         61
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:    5.102s (  962.254 blocks/s,  60.141 Mb/s)
Attachment 24: Screenshot_from_2022-05-12_11-57-44.png
Screenshot_from_2022-05-12_11-57-44.png
Attachment 25: Screenshot_from_2022-05-12_11-56-42.png
Screenshot_from_2022-05-12_11-56-42.png
Attachment 26: Screenshot_from_2022-05-12_22-32-53.png
Screenshot_from_2022-05-12_22-32-53.png
Attachment 27: Screenshot_from_2022-05-12_22-32-22.png
Screenshot_from_2022-05-12_22-32-22.png
  430   Thu May 12 00:37:52 2022 OH, AMThursday 12 May
01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
      That is now complete.
      Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc

      R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
      Will lower 2 and 4 to 200 and raise 6 and 8 to 300

      R17 2 good, 4 shakey 6 and 8 poor - attachment 2

      R18 2 and 4 at 220 and 6 and 7 at 350
      2 and 8 low deadtime, 4 and 6 could be better. - attachment 3


      Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
      Statistics at this rate - attachment 4




02:18 Statistics - attachment 7
      Temperatures - attachment 6
      Bias & leakage current - attachment 5

      Clock Status - OK
      White Rabbit - OK
      The other 3 checks currently aren't working




04:11 Statistics - attachment 10
      Temperatures - attachment 9
      Bias & leakage current - attachment 8

      Clock Status - OK
      White Rabbit - OK




06:05 Statistics - attachment 13
      Temperatures - attachment 12
      Bias & leakage current - attachment 11

      Clock Status - OK
      White Rabbit - OK
Attachment 1: R16_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      18197
          ADC data format:  144240941 ( 1037416.0 Hz)
        Other data format:    4701506 (   33814.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        651 (       4.7 Hz)
                               RESUME:        651 (       4.7 Hz)
                              SYNC100:      18715 (     134.6 Hz)
                              WR48-63:      18715 (     134.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4662774 (   33535.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:       9906 (      71.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      139.039 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               96.976
  1                1.224                0.000
  2                0.000                2.784
  3                4.445                0.000
  4                0.000              108.587
  5               78.372                0.000
  6                0.000               48.067
  7               67.687                0.000
  8                0.000              100.043
  9                0.000              102.969
 10                0.000              122.226
 11                0.000               21.170
 12                0.000                2.713
 13                0.000              116.657
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     346857       9766          0          0          0          0         47         47          0       9672          0       2994
  1   28465912       7268          0          0         10         10       3624       3624          0          0          0       3890
  2     235746    2352664          0          0          0          0        299        299          0    2352066          0        528
  3   33196911    2283279          0          0         20         20       4471       4471          0    2274297          0        644
  4     346134         72          0          0          0          0         36         36          0          0          0          4
  5   37750759       9970          0          0        309        309       4676       4676          0          0          0        836
  6     855292      17907          0          0          0          0        114        114          0      17679          0        522
  7   38520306      19374          0          0        312        312       4845       4845          0       9060          0        313
  8     312768         68          0          0          0          0         34         34          0          0          0         23
  9     342945         90          0          0          0          0         45         45          0          0          0         14
 10     176833         46          0          0          0          0         23         23          0          0          0          7
 11    1559416        390          0          0          0          0        195        195          0          0          0         62
 12    1848830        538          0          0          0          0        269        269          0          0          0         19
 13     282232         74          0          0          0          0         37         37          0          0          0         50
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   18.576s (  979.614 blocks/s,  61.226 Mb/s)
Attachment 2: R17_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       9844
          ADC data format:   78473161 ( 1119546.3 Hz)
        Other data format:    2099979 (   29959.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        470 (       6.7 Hz)
                               RESUME:        470 (       6.7 Hz)
                              SYNC100:      10105 (     144.2 Hz)
                              WR48-63:      10105 (     144.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2078829 (   29657.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       70.094 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               61.805
  1                4.886                0.000
  2                0.000                0.000
  3               17.776                0.000
  4                0.000               55.325
  5               34.711                0.000
  6                0.000               20.226
  7               27.353                0.000
  8                0.000               57.388
  9                0.000               53.302
 10                0.000               59.261
 11                0.000                7.690
 12                0.000               11.252
 13                0.000               58.364
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     137192       4907          0          0          0          0         16         16          0       4875          0          0
  1   18540829       4668          0          0         40         40       2294       2294          0          0          0          0
  2     107308    1183296          0          0          0          0        177        177          0    1182942          0          0
  3   18083643     896022          0          0        126        126       2379       2379          0     891012          0          0
  4     171162         40          0          0          0          0         20         20          0          0          0          0
  5   19343787       5154          0          0        155        155       2422       2422          0          0          0          0
  6     395908        126          0          0          0          0         63         63          0          0          0          0
  7   19401210       5206          0          0        149        149       2454       2454          0          0          0          0
  8     159836         40          0          0          0          0         20         20          0          0          0          0
  9     172050         38          0          0          0          0         19         19          0          0          0          0
 10      94357         20          0          0          0          0         10         10          0          0          0          0
 11     792883        200          0          0          0          0        100        100          0          0          0          0
 12     933311        234          0          0          0          0        117        117          0          0          0          0
 13     139685         28          0          0          0          0         14         14          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   10.115s (  973.186 blocks/s,  60.824 Mb/s)
Attachment 3: R18_19_analyse.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       9700
          ADC data format:   70825654 ( 1030660.0 Hz)
        Other data format:    8568846 (  124694.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        268 (       3.9 Hz)
                               RESUME:        268 (       3.9 Hz)
                              SYNC100:       9956 (     144.9 Hz)
                              WR48-63:       9956 (     144.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8548398 (  124396.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          2 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       68.719 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.039
  1                0.526                0.000
  2                0.044                0.000
  3               16.252                0.000
  4                0.000               41.343
  5               16.542                0.000
  6                0.000               37.115
  7                4.958                0.000
  8                0.000               62.160
  9                0.000               48.394
 10                0.000               57.328
 11                0.000               39.469
 12                0.000                0.000
 13                0.000               54.952
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     123341       4797          0          0          0          0          9          9          0       4779          0          0
  1   16458296       4022          0          0          5          5       2006       2006          0          0          0          0
  2      67098    4844172          0          0          2          2        649        649          0    4842870          0          2
  3   14877427    3705657          0          0        108        108       2346       2346          0    3700749          0          0
  4     201730         54          0          0          0          0         27         27          0          0          0          0
  5   18757776       4968          0          0        118        118       2366       2366          0          0          0          0
  6     275379         62          0          0          0          0         31         31          0          0          0          0
  7   18591077       4758          0          0         35         35       2344       2344          0          0          0          0
  8      87837         12          0          0          0          0          6          6          0          0          0          0
  9     156274         28          0          0          0          0         14         14          0          0          0          0
 10      61114         18          0          0          0          0          9          9          0          0          0          0
 11     292464         64          0          0          0          0         32         32          0          0          0          0
 12     736144        204          0          0          0          0        102        102          0          0          0          0
 13     139697         30          0          0          0          0         15         15          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   10.125s (  958.025 blocks/s,  59.877 Mb/s)
Attachment 4: 220512_0155_stats.png
220512_0155_stats.png
Attachment 5: 220512_0217_biasleakage.png
220512_0217_biasleakage.png
Attachment 6: 220512_0215_temps.png
220512_0215_temps.png
Attachment 7: 220512_0214_stats.png
220512_0214_stats.png
Attachment 8: 220512_0410_biasLeak.png
220512_0410_biasLeak.png
Attachment 9: 220512_0410_temps.png
220512_0410_temps.png
Attachment 10: 220512_0408_stats.png
220512_0408_stats.png
Attachment 11: 220512_0604_biasLeakage.png
220512_0604_biasLeakage.png
Attachment 12: 220512_0603_temps.png
220512_0603_temps.png
Attachment 13: 220512_0601_stats.png
220512_0601_stats.png
  429   Wed May 11 19:08:41 2022 CB, PPWednesday 11th May - evening shift

20:19 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

 

22:12 no beam yet.

Statistic check (screenshot attached).

Temperatures OK (screenshot attached).

Bias and leakage currents OK (same screenshot as temps).

All ADC failed calibration. Clock and white rabbit all passed.

Attachment 1: 16.png
16.png
Attachment 2: 22.png
22.png
Attachment 3: 13.png
13.png
Attachment 4: 14.png
14.png
  428   Wed May 11 07:04:13 2022 OH, NH, TDWednesday 11th May
08:00 Overnight at some point it appears that aida14 has stopped producing any ASIC data
      It is currently sending data pause requests and data push requests but nothing else - attachment 1
      telnet into aida14 and run top to see what the current state of the FEE is. AIDAExec currently using 0% CPU usage.
      FEE appears to have hung. - attachment 2
      Still passes all system wide checks.
      When performing a check load can see the tclsh process take up CPU usage

09:02 AIDA out of time sorter. Going to perform a power cycle to see if it recovers FEE14
      AIDA14 recovered but ASIC2 still missing
    
      Current working hypothesis is ASIC2 on AIDA14 is dead and that the problem with AIDA05 is either DSSD, cable or adaptor board.
      Given access would change the adapter board on AIDA05.
      Could consider installing the old AIDA05 FEE above AIDA14 and moving the cables up 1.
      Would retain the option of a speedybacktrack

      Alternatively the HEC events in AIDA05 are just above threshold. Can raise the HEC threshold
      A threshold of 0x4 reduced the rate in the HEC channels considerably
      A threshold of 0x6 removes entirely

      Note when the rate of HEC channels decreases can see the pulser in all channels of AIDA05 again. Showing that the adapter board is properly aligned

09:59 Had access so replaced the adapter card on AIDA05
      Did note that while replacing the card the right hand edge of the right hand ribbon cable was lifted slightly.
      This could also have been the cause


10:09 Restarted DAQ and there is no change to the behavior of AIDA05
      Still see events in HEC
      Again raising threshold to 0x5 in ASIC1 largely removes them. 

13.10 Attachments 3-5
      DAQ histogramming only enabled, waveforms/data transfer disabled, sampling ADCs OFF
      ASIC check load
      ASIC slow comparator all FEE64s all ASICS 0xa
      ADC data item stats

13.17 Attachments 6-9
      DAQ histogramming & data transfer enabled, waveforms disabled, sampling ADCs OFF
      ADC data item stats
      NewMerger/Tape Server

18:00 As part of tests in the afternoon it was investigated powering only one DSSD worth of FEEs at a time
      It was found that reasonable rates could be found for DSSD1 but poor rates only for DSSD2

      To try and help isolate the two detectors the jumpers 2 and 4 were removed from FEEs 13, 5 and 14 (Top, DSSD2)
      After powering on it was found there was no improvement and possible worse noise than before

      We were able to recover the rates in the 1DSSD setup so instead have removed FEE power from FEE5 and FEE14
Attachment 1: 220511_0806_aida14.png
220511_0806_aida14.png
Attachment 2: 220511_0817_aida14_top.png
220511_0817_aida14_top.png
Attachment 3: Screenshot_from_2022-05-11_13-11-09.png
Screenshot_from_2022-05-11_13-11-09.png
Attachment 4: Screenshot_from_2022-05-11_13-10-29.png
Screenshot_from_2022-05-11_13-10-29.png
Attachment 5: Screenshot_from_2022-05-11_13-09-12.png
Screenshot_from_2022-05-11_13-09-12.png
Attachment 6: Screenshot_from_2022-05-11_13-16-32.png
Screenshot_from_2022-05-11_13-16-32.png
Attachment 7: Screenshot_from_2022-05-11_13-16-24.png
Screenshot_from_2022-05-11_13-16-24.png
Attachment 8: Screenshot_from_2022-05-11_13-16-01.png
Screenshot_from_2022-05-11_13-16-01.png
Attachment 9: Screenshot_from_2022-05-11_13-15-45.png
Screenshot_from_2022-05-11_13-15-45.png
  427   Tue May 10 07:41:28 2022 NH TDTuesday 10 May
8:15am S4 CONTROLLED ACCESS to remove beam plug and check bPlas/AIDA response to beam

Reseat aida05 ribbon cable on adapter board while in...

No difference to rates, aida05 still shows hot asic01
aida14 seems seated OK, still asic02 basically empty

Noise level still high, threshold set to 0x20 (320 keV) which has decent rate

Start forwarding to MBS with 0x20 thresholds

10.12 slow comparator 0x20 -> 0xa

Attachments 1-5

Attachment 6 ADC data items 
             c. 150-180k per FEE64, c. 21Mb to disk

Attachment 7 FEE64 temps OK

Attachment 8 DSSSD bias & leakage currents OK


13.30 system checks

      N.B. aida05 ASIC temp low and pause/resume data rate high

Attachment 9 grafana monitor of DSSSD bias & leakage current & temperature OK

Attachments 10-15 spectra

Attacahments 16-19 system wide checks OK *except* WR error counter

Attachment 20 ADC data items 
             c. 150-180k per FEE64, c. 21Mb to disk

Attachment 21 FEE64 temps OK

Attachment 22 DSSSD bias & leakage currents OK

Attachments 23-29 newMerger stats

20:37 - Begin implantation test, raise threshold to 0x64 just to ensure system is stable

22:26 - No implantation test although we had some brief beam through AIDA.
      - FRS team having difficulty aligning beam still, will work overnight and try to implant tomorrow
        Will keep DAQ on 0x64 to eliminate deadtime issues for implantation
Attachment 1: Screenshot_from_2022-05-10_10-11-54.png
Screenshot_from_2022-05-10_10-11-54.png
Attachment 2: Screenshot_from_2022-05-10_10-10-09.png
Screenshot_from_2022-05-10_10-10-09.png
Attachment 3: Screenshot_from_2022-05-10_10-08-55.png
Screenshot_from_2022-05-10_10-08-55.png
Attachment 4: Screenshot_from_2022-05-10_10-08-02.png
Screenshot_from_2022-05-10_10-08-02.png
Attachment 5: Screenshot_from_2022-05-10_10-07-01.png
Screenshot_from_2022-05-10_10-07-01.png
Attachment 6: Screenshot_from_2022-05-10_10-06-30.png
Screenshot_from_2022-05-10_10-06-30.png
Attachment 7: Screenshot_from_2022-05-10_10-06-20.png
Screenshot_from_2022-05-10_10-06-20.png
Attachment 8: Screenshot_from_2022-05-10_10-06-15.png
Screenshot_from_2022-05-10_10-06-15.png
Attachment 9: Screenshot_from_2022-05-10_13-40-43.png
Screenshot_from_2022-05-10_13-40-43.png
Attachment 10: Screenshot_from_2022-05-10_13-40-01.png
Screenshot_from_2022-05-10_13-40-01.png
Attachment 11: Screenshot_from_2022-05-10_13-37-40.png
Screenshot_from_2022-05-10_13-37-40.png
Attachment 12: Screenshot_from_2022-05-10_13-37-33.png
Screenshot_from_2022-05-10_13-37-33.png
Attachment 13: Screenshot_from_2022-05-10_13-36-39.png
Screenshot_from_2022-05-10_13-36-39.png
Attachment 14: Screenshot_from_2022-05-10_13-35-06.png
Screenshot_from_2022-05-10_13-35-06.png
Attachment 15: Screenshot_from_2022-05-10_13-34-44.png
Screenshot_from_2022-05-10_13-34-44.png
Attachment 16: Screenshot_from_2022-05-10_13-31-05.png
Screenshot_from_2022-05-10_13-31-05.png
Attachment 17: Screenshot_from_2022-05-10_13-30-59.png
Screenshot_from_2022-05-10_13-30-59.png
Attachment 18: Screenshot_from_2022-05-10_13-30-52.png
Screenshot_from_2022-05-10_13-30-52.png
Attachment 19: Screenshot_from_2022-05-10_13-29-58.png
Screenshot_from_2022-05-10_13-29-58.png
Attachment 20: Screenshot_from_2022-05-10_13-29-42.png
Screenshot_from_2022-05-10_13-29-42.png
Attachment 21: Screenshot_from_2022-05-10_13-29-20.png
Screenshot_from_2022-05-10_13-29-20.png
Attachment 22: Screenshot_from_2022-05-10_13-28-48.png
Screenshot_from_2022-05-10_13-28-48.png
Attachment 23: Screenshot_from_2022-05-10_13-51-23.png
Screenshot_from_2022-05-10_13-51-23.png
Attachment 24: Screenshot_from_2022-05-10_13-51-05.png
Screenshot_from_2022-05-10_13-51-05.png
Attachment 25: Screenshot_from_2022-05-10_13-49-53.png
Screenshot_from_2022-05-10_13-49-53.png
Attachment 26: Screenshot_from_2022-05-10_13-49-41.png
Screenshot_from_2022-05-10_13-49-41.png
Attachment 27: Screenshot_from_2022-05-10_13-49-32.png
Screenshot_from_2022-05-10_13-49-32.png
Attachment 28: Screenshot_from_2022-05-10_13-49-12.png
Screenshot_from_2022-05-10_13-49-12.png
Attachment 29: Screenshot_from_2022-05-10_13-48-56.png
Screenshot_from_2022-05-10_13-48-56.png
  426   Mon May 9 08:36:23 2022 TD, NHMonday 9 May
09.32 Re-start AIDA

Attachments 1 & 2 1.8.W spectra 20us FSR

Attachments 3 & 4 1.8.L spectra
 aida10 98ch FWHM
 aida06 384 ch FWHM

Attachment 5 per FEE64 rate spectra

Attachments 6-13 system wide checks - all OK

Attachment 14 ADC data item stats
 n+n FEE64s c. 250k -> c. 370k
 aida01, 7 & 12 c. 200-400k 

Attachment 15 FEE64 temps OK

Attachment 16 DSSD bias & leakage current OK

Attachment 17 per FEE64 sys console status summary



10.15 Move triPlast Lemo-00 cabling away from aida02 DSSSD-adaptor PCB ribbon cable
      ASIC check load

Attachments 18 ADC data item stats
 no change to n+n FEE64s, very high p+n FEE64s now at lower rates

Attachments 19 & 20 1.8.W spectra 20us FSR


18.27 all ADC data rates very high
      aida05, asic #1 is again producing HEC data
      aida14, asic #2 producing very little data

Attachments 23 & 24 1.8.W spectra 20us FSR

Attachments 25 & 26 1.8.L spectra
 aida10 165ch FWHM

Attachment 27 per FEE64 rate spectra

Attachment 28 ADC data item stats

Attachment 29 FEE64 temps OK

Attachment 30 DSSD bias & leakage current OK

Attachments 21 & 22 1.8.L spectra
 aida10 65ch FWHM
 aida06 394ch FWHM

Attachment 31 Rate with threesholds at 0x64

14.00 Check 3x IC RF signals are not source of extrinsic source (c. 2MHz & c. 100kHz) observed. NT switches off all 3x RF signals - no change. 


14.30 Replace FEE64 aida05
      old d8:80:39:41:d7:cc -> d8:80:39:41:b4:0b
      old dhcpd.conf file backed up
Attachment 1: Screenshot_from_2022-05-09_09-58-10.png
Screenshot_from_2022-05-09_09-58-10.png
Attachment 2: Screenshot_from_2022-05-09_09-56-53.png
Screenshot_from_2022-05-09_09-56-53.png
Attachment 3: Screenshot_from_2022-05-09_09-54-56.png
Screenshot_from_2022-05-09_09-54-56.png
Attachment 4: Screenshot_from_2022-05-09_09-53-43.png
Screenshot_from_2022-05-09_09-53-43.png
Attachment 5: Screenshot_from_2022-05-09_09-51-41.png
Screenshot_from_2022-05-09_09-51-41.png
Attachment 6: Screenshot_from_2022-05-09_09-50-11.png
Screenshot_from_2022-05-09_09-50-11.png
Attachment 7: Screenshot_from_2022-05-09_09-50-07.png
Screenshot_from_2022-05-09_09-50-07.png
Attachment 8: Screenshot_from_2022-05-09_09-50-02.png
Screenshot_from_2022-05-09_09-50-02.png
Attachment 9: Screenshot_from_2022-05-09_09-48-13.png
Screenshot_from_2022-05-09_09-48-13.png
Attachment 10: Screenshot_from_2022-05-09_09-48-06.png
Screenshot_from_2022-05-09_09-48-06.png
Attachment 11: Screenshot_from_2022-05-09_09-48-00.png
Screenshot_from_2022-05-09_09-48-00.png
Attachment 12: Screenshot_from_2022-05-09_09-47-50.png
Screenshot_from_2022-05-09_09-47-50.png
Attachment 13: Screenshot_from_2022-05-09_09-47-27.png
Screenshot_from_2022-05-09_09-47-27.png
Attachment 14: Screenshot_from_2022-05-09_09-47-21.png
Screenshot_from_2022-05-09_09-47-21.png
Attachment 15: Screenshot_from_2022-05-09_09-47-12.png
Screenshot_from_2022-05-09_09-47-12.png
Attachment 16: Screenshot_from_2022-05-09_09-47-00.png
Screenshot_from_2022-05-09_09-47-00.png
Attachment 17: Screenshot_from_2022-05-09_09-36-29.png
Screenshot_from_2022-05-09_09-36-29.png
Attachment 18: Screenshot_from_2022-05-09_10-16-16.png
Screenshot_from_2022-05-09_10-16-16.png
Attachment 19: Screenshot_from_2022-05-09_10-17-40.png
Screenshot_from_2022-05-09_10-17-40.png
Attachment 20: Screenshot_from_2022-05-09_10-17-09.png
Screenshot_from_2022-05-09_10-17-09.png
Attachment 21: Screenshot_from_2022-05-09_10-32-52.png
Screenshot_from_2022-05-09_10-32-52.png
Attachment 22: Screenshot_from_2022-05-09_10-31-48.png
Screenshot_from_2022-05-09_10-31-48.png
Attachment 23: Screenshot_from_2022-05-09_18-16-39.png
Screenshot_from_2022-05-09_18-16-39.png
Attachment 24: Screenshot_from_2022-05-09_18-16-01.png
Screenshot_from_2022-05-09_18-16-01.png
Attachment 25: Screenshot_from_2022-05-09_18-15-32.png
Screenshot_from_2022-05-09_18-15-32.png
Attachment 26: Screenshot_from_2022-05-09_18-14-56.png
Screenshot_from_2022-05-09_18-14-56.png
Attachment 27: Screenshot_from_2022-05-09_18-13-43.png
Screenshot_from_2022-05-09_18-13-43.png
Attachment 28: Screenshot_from_2022-05-09_18-13-26.png
Screenshot_from_2022-05-09_18-13-26.png
Attachment 29: Screenshot_from_2022-05-09_18-13-16.png
Screenshot_from_2022-05-09_18-13-16.png
Attachment 30: Screenshot_from_2022-05-09_18-13-03.png
Screenshot_from_2022-05-09_18-13-03.png
Attachment 31: AnyDeskMSI_2022-05-09_21-45-12.png
AnyDeskMSI_2022-05-09_21-45-12.png
  425   Sun May 8 17:43:57 2022 NHAIDA FEE64 Layout
AIDA FEE64 layout with respect to the beam.
Attachment 1: AIDA_Adapter_Configuration(1).png
AIDA_Adapter_Configuration(1).png
  424   Sun May 8 15:10:45 2022 TDSunday 8 May
DSSSD bias and FEE64 power OFF during Ge install



Photographs of AIDA (triple) + DEGAS setup as of Sunday pm.

Attachments 1-4 side/front/oblique views - note AIDA snout droops and twists

Attachments 5 & 6 - position of the AIDA stand vertical adjustment threads (brass) 

The asymmetry upstream-downstream pre-dates this setup. To lift the end of the snout clear of the of lower DEGAS detectors all 3x 1/4 turns clockwise (viewed from above) were applied to each of the four threads.
Attachment 1: 20220508_160103.jpg
20220508_160103.jpg
Attachment 2: 20220508_160115.jpg
20220508_160115.jpg
Attachment 3: 20220508_160128.jpg
20220508_160128.jpg
Attachment 4: 20220508_160140.jpg
20220508_160140.jpg
Attachment 5: 20220508_160720.jpg
20220508_160720.jpg
Attachment 6: 20220508_160804.jpg
20220508_160804.jpg
  423   Sat May 7 13:27:39 2022 TDAIDA configuration
Adaptor PCB link settings

n+n FEE64s

aida02, 4, 6 & 8 - LK1 fitted, HV Lemo-00 not connected - left/right wafer n+n side bias ring -> gnd

p+n FEE64s

   top - aida09, 1, 10 & aida13, 5, 14 - LK2 & LK4 fitted - front/back FP -> gnd
bottom - aida11, 3, 12 & aida15, 7, 16 - LK2 & LK4 fitted - front/back FP -> gnd

   top - aida01, 5 - LK3 fitted - middle wafer n+n side bias ring -> gnd
bottom - aida03, 7 - LK3 fitted - DSSSD PCB gnd -> gnd

AIDA PSU

   top 1-2, 3-4, 9-10, 11-12
bottom 5-6, 7-8, 13-14, 15-16


ASIC settings 2021Apr29-13-16-00
     slow comparator 0x64 -> 0xa

BNC PB-5 settings
    amplitude 1.000V
    attenuation x1
    tail pulse
    tau_d 1ms
    polarity +
    frequency 25Hz
  422   Sat May 7 09:55:53 2022 TDSaturday 7 May
Attachments 1 & 2 per FEE64 1.8.L spectra 
 pulser peak widths aida01 121 ch FWHM, aida02 517 ch FWHM

Attachments 3 & 4 per FEE64 1.8.W spectra 20us FSR

Attachment 5 per FEE64 rate spectra

Attachment 6 - grafana DSSSD bias/leakage current for previous 7 days showing effect of day/night temp variations

Attachments 7 & 8 - merger & tape server c. 14Mb/s to disk - no errors reported by servers

Attachments 9-12 - system wide checks - all OK

Attachment 13 ADC data item stats

Attachment 14 FEE64 temps OK

Attachment 15 DSSSD bias & leakage currents OK

16.15 aida05 zero ADC data items stats
      merger restart, DAQ STOP/Go cycle to recover


      activity is S4 area near AIDA DSSSDs as install of DEGAS detectors commences

19.50 DEGAS install continues

      DSSSD bias and FEE64 power OFF
 
Attachment 1: Screenshot_from_2022-05-07_10-53-56.png
Screenshot_from_2022-05-07_10-53-56.png
Attachment 2: Screenshot_from_2022-05-07_10-52-54.png
Screenshot_from_2022-05-07_10-52-54.png
Attachment 3: Screenshot_from_2022-05-07_10-51-33.png
Screenshot_from_2022-05-07_10-51-33.png
Attachment 4: Screenshot_from_2022-05-07_10-51-06.png
Screenshot_from_2022-05-07_10-51-06.png
Attachment 5: Screenshot_from_2022-05-07_10-47-13.png
Screenshot_from_2022-05-07_10-47-13.png
Attachment 6: Screenshot_from_2022-05-07_10-46-40.png
Screenshot_from_2022-05-07_10-46-40.png
Attachment 7: Screenshot_from_2022-05-07_10-46-26.png
Screenshot_from_2022-05-07_10-46-26.png
Attachment 8: Screenshot_from_2022-05-07_10-46-16.png
Screenshot_from_2022-05-07_10-46-16.png
Attachment 9: Screenshot_from_2022-05-07_10-45-34.png
Screenshot_from_2022-05-07_10-45-34.png
Attachment 10: Screenshot_from_2022-05-07_10-45-28.png
Screenshot_from_2022-05-07_10-45-28.png
Attachment 11: Screenshot_from_2022-05-07_10-45-23.png
Screenshot_from_2022-05-07_10-45-23.png
Attachment 12: Screenshot_from_2022-05-07_10-45-17.png
Screenshot_from_2022-05-07_10-45-17.png
Attachment 13: Screenshot_from_2022-05-07_10-45-08.png
Screenshot_from_2022-05-07_10-45-08.png
Attachment 14: Screenshot_from_2022-05-07_10-43-29.png
Screenshot_from_2022-05-07_10-43-29.png
Attachment 15: Screenshot_from_2022-05-07_10-43-14.png
Screenshot_from_2022-05-07_10-43-14.png
  421   Fri May 6 09:31:41 2022 TD NHFriday 6 May
10:30 DAQ appears to have stopped

      Attachment 2 - DSSSD bias and leakage current OK

      Attachments 1, 3-5 - zero rate adc data items all FEE64s - note aida08 adc data item counter zero

      RESET/SETUP/GO aida08 *only*
      DAQ STOP *all* FEE64s


      Attachment 6 - FEE64 temperatures OK

      Attachments 7-23 - system wide checks - all OK

10:48 DAQ GO *all* FEE64s
      all spectra zero'd

      attachment 29 - ADC data items
       n+n FEE64s aida02, 4, 6 & 8 c. 250k
       p+n FEE64s aida07, 9 & 12 c. 100-150k, all other FEE64s < 100k

      attachments 26 & 27 1.8.L spectra
       aida01 peak width 112ch FWHM
       aida02 peak width 485 ch FWHM

      attachments 24 & 25 1.8.W spectra 20us FSR
      
      attachment 30 - aida05 1*H spectra - aida05 asic#1 is producing HEC data

      attachment 31 - aida05 also producing disc data

      attachments 32 & 33 merger and tape server OK - 14MB/s to disk - no data to MBS currently? 


18.36 Disconnect ribbon cabling from adaptor PCB - continue to observe HEC events from aida05 asic#1
      Disconnect test inputs from adaptor PCB - continue to observe HEC events from aida05 asic#1

      Conclude aida05 asic#1 has probably failed in operation and should be replaced. Other FEE64s are currently in CRYRING hall and measurement is in progress - ML says we should be able to access Monday.
Attachment 1: Screenshot_from_2022-05-06_10-27-27.png
Screenshot_from_2022-05-06_10-27-27.png
Attachment 2: Screenshot_from_2022-05-06_10-26-59.png
Screenshot_from_2022-05-06_10-26-59.png
Attachment 3: Screenshot_from_2022-05-06_10-26-33.png
Screenshot_from_2022-05-06_10-26-33.png
Attachment 4: Screenshot_from_2022-05-06_10-25-44.png
Screenshot_from_2022-05-06_10-25-44.png
Attachment 5: Screenshot_from_2022-05-06_10-24-54.png
Screenshot_from_2022-05-06_10-24-54.png
Attachment 6: Screenshot_from_2022-05-06_10-42-18.png
Screenshot_from_2022-05-06_10-42-18.png
Attachment 7: Screenshot_from_2022-05-06_10-41-06.png
Screenshot_from_2022-05-06_10-41-06.png
Attachment 8: Screenshot_from_2022-05-06_10-41-00.png
Screenshot_from_2022-05-06_10-41-00.png
Attachment 9: Screenshot_from_2022-05-06_10-40-54.png
Screenshot_from_2022-05-06_10-40-54.png
Attachment 10: Screenshot_from_2022-05-06_10-40-09.png
Screenshot_from_2022-05-06_10-40-09.png
Attachment 11: Screenshot_from_2022-05-06_10-39-07.png
Screenshot_from_2022-05-06_10-39-07.png
Attachment 12: Screenshot_from_2022-05-06_10-38-58.png
Screenshot_from_2022-05-06_10-38-58.png
Attachment 13: Screenshot_from_2022-05-06_10-38-35.png
Screenshot_from_2022-05-06_10-38-35.png
Attachment 14: Screenshot_from_2022-05-06_10-38-21.png
Screenshot_from_2022-05-06_10-38-21.png
Attachment 15: Screenshot_from_2022-05-06_10-38-16.png
Screenshot_from_2022-05-06_10-38-16.png
Attachment 16: Screenshot_from_2022-05-06_10-38-10.png
Screenshot_from_2022-05-06_10-38-10.png
Attachment 17: Screenshot_from_2022-05-06_10-38-02.png
Screenshot_from_2022-05-06_10-38-02.png
Attachment 18: Screenshot_from_2022-05-06_10-37-52.png
Screenshot_from_2022-05-06_10-37-52.png
Attachment 19: Screenshot_from_2022-05-06_10-37-46.png
Screenshot_from_2022-05-06_10-37-46.png
Attachment 20: Screenshot_from_2022-05-06_10-37-39.png
Screenshot_from_2022-05-06_10-37-39.png
Attachment 21: Screenshot_from_2022-05-06_10-37-31.png
Screenshot_from_2022-05-06_10-37-31.png
Attachment 22: Screenshot_from_2022-05-06_10-36-24.png
Screenshot_from_2022-05-06_10-36-24.png
Attachment 23: Screenshot_from_2022-05-06_10-35-52.png
Screenshot_from_2022-05-06_10-35-52.png
Attachment 24: Screenshot_from_2022-05-06_10-56-31.png
Screenshot_from_2022-05-06_10-56-31.png
Attachment 25: Screenshot_from_2022-05-06_10-55-56.png
Screenshot_from_2022-05-06_10-55-56.png
Attachment 26: Screenshot_from_2022-05-06_10-55-13.png
Screenshot_from_2022-05-06_10-55-13.png
Attachment 27: Screenshot_from_2022-05-06_10-53-53.png
Screenshot_from_2022-05-06_10-53-53.png
Attachment 28: Screenshot_from_2022-05-06_10-51-00.png
Screenshot_from_2022-05-06_10-51-00.png
Attachment 29: Screenshot_from_2022-05-06_10-50-25.png
Screenshot_from_2022-05-06_10-50-25.png
Attachment 30: Screenshot_from_2022-05-06_11-09-15.png
Screenshot_from_2022-05-06_11-09-15.png
Attachment 31: Screenshot_from_2022-05-06_11-02-33.png
Screenshot_from_2022-05-06_11-02-33.png
Attachment 32: Screenshot_from_2022-05-06_11-19-48.png
Screenshot_from_2022-05-06_11-19-48.png
Attachment 33: Screenshot_from_2022-05-06_11-19-27.png
Screenshot_from_2022-05-06_11-19-27.png
  420   Wed May 4 16:50:04 2022 NHAIDA 4th May
bPlas has been mostly cabled in now

Reseat LAN cable (AIDA Switch -> AIDA-3) and reachieve 1 gbit/s speed as needed.

CHeck DAQ with MBS, working well (16 MB/s). Performance seems better than before on MBS side too

Note implant counts in aida05, observable in rate spectra (fig 1)
Also a few early channels in aida10 seem empty

ASIC check-loads didn't fix
Try powercycle... also didn't fix

Tomorrow will inspect adapter board and ribbon cable, perhaps it is loose.

Absolute rates otherwise seem OK (fig 2) aida01 and aida07 are both very hot.
aida01 is usually quieter but aida07 has always been observed running fast
Attachment 1: Screenshot_20220504_174929.png
Screenshot_20220504_174929.png
Attachment 2: Screenshot_20220504_175201.png
Screenshot_20220504_175201.png
  419   Tue May 3 14:15:17 2022 NHAIDA Updates 2 May
PAUSE/RESUME stats are dominated by 2x FEE64s - aida07 and aida16

These FEE64s also have the highest rates of discriminator data?

> The snout endcap has been sealed with new copper tape over the red tape
> 
> Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
> It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive
> 
> 
> 
> -
> 
> the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
> When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s
> 
> -
> 
> Preliminary alpha background analysis (over weekend, 90 hours statistics)
> 
> Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04
> 
> ======================================
> AIDA Unpacker Analysis
> --------------------------------------
> First event: Thu Apr 28 19:50:20 2022
> Last event : Mon May  2 14:04:03 2022
> Duration   : 324823 seconds
> --------------------------------------
> MBS Events   (M):     912899 (2 Hz)
> AIDA Words   (W): 7472078760 (-3441 Hz)
> ADC Words    (A): 2742871865 (-4778 Hz)
> -- Decays    (D): 2728717223 (-4821 Hz)
> -- Implants  (I):   14154642 (43 Hz)
> Info Words   (F): 4729206895 (1 kHz)
> -- SYNC48    (S):    2093328 (6 Hz)
> -- SYNC63    (s):    2093328 (6 Hz)
> -- PAUSE     (P):    3823161 (11 Hz)
> -- RESUME    (R):    3823161 (11 Hz)
> -- Discrim   (D): 1390520565 (4 kHz)
> -- Scaler    (C): 3326853352 (3 kHz)
> -- Unknown   (U):          0 (0 Hz)
> Timewarps    (T):         20 (0 Hz)
> Dead Time    (X): 5024.43 s (1.54682%)
> ======================================
> ======================================
> Individual FEE Overview
> ======================================
>  FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
>    1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
>    2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
>    3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
>    4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
>    5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
>    6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
>    7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
>    8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
>    9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
>   10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
>   11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
>   12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
>   13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
>   14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
>   15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
>   16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9
  418   Mon May 2 13:08:49 2022 NHAIDA Updates 2 May
The snout endcap has been sealed with new copper tape over the red tape

Observe significant drop in leakage current on DSSD #2 down to 7 uA (about 0.5 uA above DSSD #1) as it looked when the black cloth was on
It is possible more sealing needs to be done when bPlas is biased as bPlas is more sensitive



-

the AIDA LAN network interface is only running at 100 MBit/s - this will be problematic during the run
When 2A is free will try to reconnect cable to see if it returens to 1 Gbit/s

-

Preliminary alpha background analysis (over weekend, 90 hours statistics)

Cause of sporadic PAUSE/RESUME is not clear as rate is nearly always 0 outside of aida02/aida04

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Apr 28 19:50:20 2022
Last event : Mon May  2 14:04:03 2022
Duration   : 324823 seconds
--------------------------------------
MBS Events   (M):     912899 (2 Hz)
AIDA Words   (W): 7472078760 (-3441 Hz)
ADC Words    (A): 2742871865 (-4778 Hz)
-- Decays    (D): 2728717223 (-4821 Hz)
-- Implants  (I):   14154642 (43 Hz)
Info Words   (F): 4729206895 (1 kHz)
-- SYNC48    (S):    2093328 (6 Hz)
-- SYNC63    (s):    2093328 (6 Hz)
-- PAUSE     (P):    3823161 (11 Hz)
-- RESUME    (R):    3823161 (11 Hz)
-- Discrim   (D): 1390520565 (4 kHz)
-- Scaler    (C): 3326853352 (3 kHz)
-- Unknown   (U):          0 (0 Hz)
Timewarps    (T):         20 (0 Hz)
Dead Time    (X): 5024.43 s (1.54682%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |          W          A          D          I          F          S          s     P     R          D     C     U     T X
   1 |       9593       6985       6628        357       2608          2          2     0     0       2604     0     0     0 0
   2 |   31134221   31115668   31115166        502      18553       8183       8183     0     0       2187     0     0     0 0
   3 |  717765720      17634      17218        416  717748086     216426     216426     0     0       1653 717313581     0     0 0
   4 | 3467468322 2684820687 2684820157        530  782647635    1293089    1293089     0     0   62747129 717314328     0    20 0
   5 |     177575     108631     107352       1279      68944         21         21    20    20      68862     0     0     0 0.00102142
   6 |    8371118    8361524    8358540       2984       9594       1350       1350    23    23       6848     0     0     0 10.6775
   7 | 1396478942    4860615      57719    4802896 1391618327     346048     346048 1286994 1286994  446838393 941513850     0     0 3071.3
   8 |  954933629    3978543    3975110       3433  950955086     118195     118195     5     5       7093 950711593     0     0 5.47318
   9 |       9877       7923       7429        494       1954          4          4     0     0       1946     0     0     0 0
  10 |       8076       6236       5917        319       1840          4          4     0     0       1832     0     0     0 0
  11 |      26792      24399      24029        370       2393          3          3     0     0       2387     0     0     0 0
  12 |      18271      16025      15382        643       2246          6          6     0     0       2234     0     0     0 0
  13 |    1094646      51444      48305       3139    1043202        195        195     1     1    1042810     0     0     0 0.179207
  14 |    7314376     152823      84786      68037    7161553        906        906 18158 18158    7123425     0     0     0 32.882
  15 |     438092      22117      20290       1827     415975         56         56   251   251     415361     0     0     0 4.91581
  16 |  886829510    9320611      53195    9267416  877508899     108840     108840 2517709 2517709  872255801     0     0     0 4970.9
Attachment 1: elog_temps1.png
elog_temps1.png
Attachment 2: alpha_1d.png
alpha_1d.png
  417   Mon May 2 09:52:20 2022 OHShifter checklist 2022
Every 30 minutes:

- Reload statistics (Web browser tab screen 1, workspace 3)
    - Are all FEEs still showing a rate?
    - If one shows 0 after multiple reloads Call Oscar or Tom.

- IGNORE FOR TIME BEING Options database monitor (Top middle terminal screen 2, workspace 3)
    - Has the message changed since the last check?
    - If so copy the message and place in elog

- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 5)
    - If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
    - Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 3)
        - Ensure "Act on all FEE" and "Act on all asic" selected
        - Click on drop down menu in the bottom of the screen and select check ASIC control
        - It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
        - A pop up will appear check that all options say ok
        - Has the implant rate gone to a more normal amount?
            - If not feel free to call

Every two hours:

- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 3)
     - Compare to the previous screenshot to check if anything has drastically changed

- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 3)
     - Do they all appear stable?

- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 3)

- Perform the five system wide checks (Web browser tab screen 1, workspace 3)
     - Ignore the system wide checks for ADC calibration (ADCs are turned off so they will all fail)
     - Check memory information also fails.
     - Make a note in the elog of which tests are successful
     - If failed copy the text from the white box into the elog and make a note of it

- Enter the screenshots into the elog

- Update the leakage current spreadsheet (Web browser, screen 2, workspace 5)
  416   Thu Apr 28 18:45:09 2022 NHAlpha Background Run
Runs are in /TapeData/BGAPR22 located on 8TB disk "SecondDrive" (N.B. 2A experiment is using ThirdDrive)

Old runs and test files were removed from disk. Left is an unknown S481 folder (fibre implant test?) and Patrick
chown to npg:npg as uid changed from old SL6 to CentOS 7

Bias DSSDs, leakage current higher than before (maybe the sheet isn't covering the snout)
Pulser set to OFF
ASIC Thresholds 0x64, rates mostly 0 (good)
Edge strips on aida02 and aida06 show non zero, and lots in aida04 asic 1 are high, will recheck connector tomorrow in hope

Connecting to merger and rate is 60k items/sec with random jumps to 400k items/sec
Start xfer to tape server: Rate approx 100 KB/s (360 MB per hour, 2.88 GB per 8 hours)

19:51 - Will leave overnight running to collect alpha

Fri 29 Apr

14:38 - Run paused for MBS tests with FRS

16:50 - Resume alpha collection over weekend
Pulser OFF, 0x64 thresholds
Attachment 1: AnyDeskMSI_2022-04-28_19-47-33.png
AnyDeskMSI_2022-04-28_19-47-33.png
Attachment 2: AnyDeskMSI_2022-04-28_19-47-56.png
AnyDeskMSI_2022-04-28_19-47-56.png
Attachment 3: AnyDeskMSI_2022-04-28_19-48-13.png
AnyDeskMSI_2022-04-28_19-48-13.png
  415   Wed Apr 27 15:47:46 2022 NH, HA, SA, CJAIDA 27 Apr 22
Remount snout in S4 (photos to come) 
Alignment with DEGAS OK but a bit short...  DEGAS platform will be brought 31 mm closer to AIDA by rail adjustment

Can mount bPlas booster boards on old ITEM frames and also partly on AIDA snout (isolated)

-

Reconnect all adapter boards and grounds, bias AIDA...

HV OK. We still have a light leak at the snout endcap. "Fixed" with sheet for now

HV leakage currents figure 1

Start AIDA DAQ and check rates.. all seem OK. aida07 is very noisy other p+n good
Notice some issues in aida04, check and ribbon cable slightly loose on one side of adapter
No change to other FEEs.
Aida07 was inspected but nothing obviously wrong

Rates figure 2
Rate spectra figure 3
Wavesform figures 4 (p+n), 5 (n+n)
Pulser peak withs
 p+n figure 6, width ~  70 channels (50 keV)
 n+n figure 7, width ~ 570 channels (400 keV)

n.b. double peak in aida14 is an artifact from an uncleared histogram

DSSDs debiased to allow work on bPlas cabling without tripping DSSDs
Attachment 1: 2022-04-27_16-42-37_HV.png
2022-04-27_16-42-37_HV.png
Attachment 2: 022-04-27_16-39-58_Rates.png
022-04-27_16-39-58_Rates.png
Attachment 3: 2022-04-27_16-40-23_RatesSpectra.png
2022-04-27_16-40-23_RatesSpectra.png
Attachment 4: 2022-04-27_16-41-46_pnwaves.png
2022-04-27_16-41-46_pnwaves.png
Attachment 5: 2022-04-27_16-42-20_nn_waves.png
2022-04-27_16-42-20_nn_waves.png
Attachment 6: 2022-04-27_16-43-30_pn_pulser_70ch.png
2022-04-27_16-43-30_pn_pulser_70ch.png
Attachment 7: 2022-04-27_16-44-18._nn_pulser_568ch.png
2022-04-27_16-44-18._nn_pulser_568ch.png
  414   Fri Apr 8 08:59:42 2022 OH, NHFriday 8th April
09:59 Noticed that the right hand ribbon cable for aida07 was offset vertically by 1. i.e. Only 1 of the 2 rows of pins in. Unbiassed detectors and reconnected.
      Rebiassed detectors - leakage current on DSSD2 increased to 9.96uA - Attachment 1

      Power cycling FEEs

      Following restart the FEEs show some improvement in rates - attachment 2
      Rate spectra - attachment 3
      Still issues with aida11 in the rate spectra (Likely a misaligned ERNI connector)



      Pulser on rate spectra - attachment 4
      Pulser off rate spectra - attachment 5
      We can see that with the pulser on there are events in the channels in aida01 and aida11
      With the pulser off these channels disappear.
      ERNI connector is then likely aligned the issue is more likely the DSSD ribbon cable is misaligned.

      Taking bias off to investigate.


10:39 Could not observe anything out of alignement about either the connectors or the adaptor boards
      Note that after pushing on the connector AIDA01 had largely recovered

13:30 Top of snout taken off while mounted in S4. It was observed that one of the ribbon cables for aida11 was off by 1 row.
      This was corrected in situ and the snout was put back together and made light tight again.
      Oddly leakage current still the same - DSSD2 has come down though - attachment 6
      FEEs powercycled for new tests

      Half the channels in aida11 had appeared again
      Took apart once more and found the other header also off by one row which has been corrected

      FEEs powercycled for new test

      All of aida11 now showing - attachment 7
      Statistics at 0xa shown in attachment 8
      All p+n for DSSD1 are good (aida01, 03, 09, 10, 11 and 12)
      p+n for DSSD2 are so so (aida05, 07, 13, 14, 15 and 16)
      All n+n 2, 4, 6 and 8 are poor still


14:50 p+n waveforms - attachment 9
      n+n waveforms - attachment 10

      Current jumper configuration
      LK1 (Bias ground) on FEEs 6 and 8
      LK2&4 on all p+n
      LK3 on aida03 and aida07 (Bottom middle)

15:00 Investigated whether there was a light leak. Placed black material over snout.
      Leakage on DSSD2 dropped to 5.6 - attachment 11
      There was a light leak
      Stats have decreased slightly - attachment 12

16:00 Move raspberry pis atop 5mm alminimum plate to keep away from FEE64 PSUs
      No change

16:30 Remove pulser network from all FEEs 
      Rates get noticably worse in many FEEs... why?
      Stats - attachment 13

      Possible ideas include braid touching things it shouldn't (Adapter PCBs?) 
Attachment 1: 220408_1000_bias.png
220408_1000_bias.png
Attachment 2: 220409_1010_Stats.png
220409_1010_Stats.png
Attachment 3: 220409_1011_rate.png
220409_1011_rate.png
Attachment 4: 220408_1358_bias.png
220408_1358_bias.png
Attachment 5: 220408_1444_rate.png
220408_1444_rate.png
Attachment 6: 220408_1444_stats.png
220408_1444_stats.png
Attachment 7: 220408_1449_layout7.png
220408_1449_layout7.png
Attachment 8: 220408_1451_layout8.png
220408_1451_layout8.png
Attachment 9: 220308_1459_bias.png
220308_1459_bias.png
Attachment 10: 220408_1503_stats_post_light.png
220408_1503_stats_post_light.png
Attachment 11: 2022-04-08_16-26-30_Stats_No_Pulser.png
2022-04-08_16-26-30_Stats_No_Pulser.png
  413   Thu Apr 7 12:42:14 2022 OH, NH, TDThursday 7 April
Re-assembled triple snout re-installed on the AIDA stand - DSSSD ribbon cables and grounds re-connected.
FEE64 adaptor PCB LKs per https://elog.ph.ed.ac.uk/DESPEC/409

Attachments 1 & 2 1.8.L spectra
  p+n FEE64 aida10 pulser peak width 97 ch FWHM
  n+n FEE64 aida04 pulser peak width 550 ch FWHM

Attachment 3 rate spectra
  missing channels aida01, aida07 & aida11 - misaligned ERNI and/or Samtec connectors?

Attachments 4 & 5 1.8.W spectra 

Attachment 6 ADC data item rates
 aida10 OK, all others c. 300-400k

Attachment 7 FEE64 temperatures OK

Attachment 8 DSSSD bias & leakage currents OK
Attachment 1: Screenshot_from_2022-04-07_13-40-55.png
Screenshot_from_2022-04-07_13-40-55.png
Attachment 2: Screenshot_from_2022-04-07_13-40-02.png
Screenshot_from_2022-04-07_13-40-02.png
Attachment 3: Screenshot_from_2022-04-07_13-37-17.png
Screenshot_from_2022-04-07_13-37-17.png
Attachment 4: Screenshot_from_2022-04-07_13-35-43.png
Screenshot_from_2022-04-07_13-35-43.png
Attachment 5: Screenshot_from_2022-04-07_13-35-23.png
Screenshot_from_2022-04-07_13-35-23.png
Attachment 6: Screenshot_from_2022-04-07_13-34-34.png
Screenshot_from_2022-04-07_13-34-34.png
Attachment 7: Screenshot_from_2022-04-07_13-34-12.png
Screenshot_from_2022-04-07_13-34-12.png
Attachment 8: Screenshot_from_2022-04-07_13-33-53.png
Screenshot_from_2022-04-07_13-33-53.png
  412   Wed Apr 6 10:09:21 2022 OH, NH, TDWednesday 6 April
DSSSD bias OFF. DAQ STOPped

Attachment 6 FEE64 temps OK

Attachment 1-5 system wide checks OK 
Attachment 1: Screenshot_from_2022-04-06_11-12-59.png
Screenshot_from_2022-04-06_11-12-59.png
Attachment 2: Screenshot_from_2022-04-06_11-12-48.png
Screenshot_from_2022-04-06_11-12-48.png
Attachment 3: Screenshot_from_2022-04-06_11-12-38.png
Screenshot_from_2022-04-06_11-12-38.png
Attachment 4: Screenshot_from_2022-04-06_11-10-06.png
Screenshot_from_2022-04-06_11-10-06.png
Attachment 5: Screenshot_from_2022-04-06_11-09-57.png
Screenshot_from_2022-04-06_11-09-57.png
Attachment 6: Screenshot_from_2022-04-06_11-09-29.png
Screenshot_from_2022-04-06_11-09-29.png
  411   Tue Apr 5 12:46:52 2022 TDMSL type BB18(DS)-1000 triple - cable lengths
OH/NH disassembling current AIDA triple snout and DSSSD stack.

Length of ribbon cables (including connector) outside snout as follows

p+n junction side 9cm & 15cm
n+n ohmic side 10.5cm & 16cm 


Update
p+n junction side: 6.5cm, 10.5 cm (used  7.5, 11.5 for 1cm slack)
n+n ohmic side:  11.0 cm, 15.0 cm (used 12.0, 16.0 for 1cm slack)
Measurement from SAMTEC connector to exit from the rubber seals in AIDA snout.
  410   Tue Apr 5 08:52:16 2022 TDTuesday 5 April
DSSSD bias OFF. DAQ STOPped

Attachment 1 FEE64 temps OK

Attachment 2 system wide checks OK *except* WR decoder (attachment 2) and FPGA ts errors no longer works

Attachment 6 System wide check - 'Synchronise ASIC clocks'

Attachment 5 System wide check - check ADC calibration - all 16x FEE64s fail

Attachment 4 System wide check - WR decoder status - aida09 WR status 0x10 clears

Repeat Synchronsie ASIC clocks

Attachment 3 System wide check - WR decoder status - aida05 WR status 0x10 clears
             
Attachment 1: Screenshot_from_2022-04-05_09-52-42.png
Screenshot_from_2022-04-05_09-52-42.png
Attachment 2: Screenshot_from_2022-04-05_10-05-59.png
Screenshot_from_2022-04-05_10-05-59.png
Attachment 3: Screenshot_from_2022-04-05_16-10-31.png
Screenshot_from_2022-04-05_16-10-31.png
Attachment 4: Screenshot_from_2022-04-05_16-09-38.png
Screenshot_from_2022-04-05_16-09-38.png
Attachment 5: Screenshot_from_2022-04-05_16-09-30.png
Screenshot_from_2022-04-05_16-09-30.png
Attachment 6: Screenshot_from_2022-04-05_16-08-44.png
Screenshot_from_2022-04-05_16-08-44.png
  409   Mon Apr 4 09:19:49 2022 TDMonday 4 April
AIDA adaptor PCB config - all LKs removed *except*

aida04 & aida08 LK1

aida02 & aida07 LK2, LK3 & LK4

aida09, aida01, aida10, aida11, aida15, aida12 & aida16 LK2 & LK4



10.05 check all ASICs all FEE64s
      zero histograms

Attachment 1 per FEE64 rate spectra

Attachment 2 per FEE64 p+n 1.8.L spectra - aida10 peak width 79 ch FWHM

Attachment 3 per FEE64 p+n 1.8.W spectra 20us FSR

Attachment 4 per FEE64 n+n 1.8.W spectra 20us FSR

Attachments 5-8 - system wide checks - all OK *except*

  FPGA timestamp errors no longer works
  aida05 & aida09 report WR decoder error 0x10

Attachment 9 ADC data item stats

Attachment 10 FEE64 temps OK

Attachment 11 DSSSD bias & leakage current OK

Attachment 12 per FEE64 n+n 1.8.L spectra - aida08 peak width 369 ch FWHM


14.25 DAQ STOP
      DSSSD bias OFF

      AIDA triple snout removed from AIDA stand and moved to table by aida-gsi (Messeheute)
Attachment 1: Screenshot_from_2022-04-04_10-18-36.png
Screenshot_from_2022-04-04_10-18-36.png
Attachment 2: Screenshot_from_2022-04-04_10-17-30.png
Screenshot_from_2022-04-04_10-17-30.png
Attachment 3: Screenshot_from_2022-04-04_10-14-53.png
Screenshot_from_2022-04-04_10-14-53.png
Attachment 4: Screenshot_from_2022-04-04_10-14-24.png
Screenshot_from_2022-04-04_10-14-24.png
Attachment 5: Screenshot_from_2022-04-04_10-13-09.png
Screenshot_from_2022-04-04_10-13-09.png
Attachment 6: Screenshot_from_2022-04-04_10-12-41.png
Screenshot_from_2022-04-04_10-12-41.png
Attachment 7: Screenshot_from_2022-04-04_10-12-33.png
Screenshot_from_2022-04-04_10-12-33.png
Attachment 8: Screenshot_from_2022-04-04_10-12-26.png
Screenshot_from_2022-04-04_10-12-26.png
Attachment 9: Screenshot_from_2022-04-04_10-11-09.png
Screenshot_from_2022-04-04_10-11-09.png
Attachment 10: Screenshot_from_2022-04-04_10-10-58.png
Screenshot_from_2022-04-04_10-10-58.png
Attachment 11: Screenshot_from_2022-04-04_10-09-37.png
Screenshot_from_2022-04-04_10-09-37.png
Attachment 12: Screenshot_from_2022-04-04_10-29-08.png
Screenshot_from_2022-04-04_10-29-08.png
  408   Sun Apr 3 10:56:11 2022 TDSunday 3 April
all system wide checks OK *except* aida10 failed ADC calibration

Attachment 1 ADC data item stats
Attachment 2 FEE64 temps OK
Attachment 3 DSSSD bias & leakage currents OK





DSSSD bias to left (looking upstream) wafer *only*
      FEE64s 10 & 12, 14 & 16

Attachments 4 & 5 1.8.W spectra 20us FSR

Attachment 6 ADC data item stats

Attachment 7 DSSSD bias & leakage current




ASIC check all FEE64s

DSSSD bias to middle (looking upstream) wafer *only*
      FEE64s 1 & 3, 5 & 7

Attachments 8 & 9 1.8.W spectra 20us FSR

Attachment 10 ADC data item stats

Attachment 11 DSSSD bias & leakage current





ASIC check all FEE64s

DSSSD bias to right (looking upstream) wafer *only*
      FEE64s 9 & 11, 13 & 15

Attachments 12 & 13 1.8.W spectra 20us FSR

Attachment 14 ADC data item stats

Attachment 15 DSSSD bias & leakage current



Conclusions

ADC data item stats -> 300-400k

n+n 1.8.W spectra show large signal excursions

p+n 1.8.W spectra for biased Si wafers OK - unbiased Si wafers show large signal excursions


DSSSD#1 leakage currents 1.850+2.220+1.990=6.06uA    cf. 5.095uA with all 3 wafers biased 
DSSSD#2 leakage currents 1.510+1.580+2.350=5.44uA    cf. 5.235uA 


Right wafer (looking upstream)
             
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -1.135 -1.475 -1.645 -1.750 -1.805
DSSSD#2 I_l(uA) -1.625 -2.020 -2.205 -2.320 -2.360

Middle wafer (looking upstream)
          
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -1.540 -1.870 -2.035 -2.150 -2.210
DSSSD#2 I_l(uA) -0.985 -1.285 -1.465 -1.575 -1.615

Left wafer (looking upstream)
          
Bias(V)         -20    -40    -60    -80    -100
DSSSD#1 I_L(uA) -0.735 -0.915 -1.005 -1.080 -1.110
DSSSD#2 I_l(uA) -0.865 -1.205 -1.400 -1.510 -1.560

Bias versus leakage cuurent characteristic of each wafer of each DSSSD indicates diode behaviour.

Note LK1 not fitted - presumably LK3 of mid/top p+n FEE64 adaptor PCBs (aida01 and aida05) connecting n+n bias to gnd is sufficient to establish circuit.




15.50 Re-connect bias to all 3x Si wafers p+n junction side bias rings

ASIC check all FEE64s

Attachments 16 & 17 1.8.W spectra 20us FSR

Attachment 18 ADC data item stats

Attachment 19 DSSSD bias & leakage current

Conclusion - status quo ante restored



16.40 Remove aida01 & aida05 LK3, add aida04 & aida08 LK1

ASIC check all FEE64s

Attachments 20 & 21 1.8.W spectra 20us FSR

Attachment 22 ADC data item stats

Attachment 23 DSSSD bias & leakage current

Conclusion - no change



17.00 Add Thorn-Lab HV filters DSSSD#1 & #2


ASIC check all FEE64s

Attachments 24 & 25 1.8.W spectra 20us FSR

Attachment 26 ADC data item stats

Attachment 27 DSSSD bias & leakage current

Conclusion - aida09-aida12 no change, aida13-aida16 -> 300-400k
             I do not know why there's a difference between DSSSD#1 & #2 but more important point (probably) no improvement aida09-aida12
             Thorn Lab HV filters removed - status quo ante restored


18.02 Remove LK2 & LK4 all top p+n adaptor PCBs (aida13, 5, 14, 9, 1, & 10)

ASIC check all FEE64s

Attachments 28 & 29 1.8.W spectra 20us FSR

Attachment 30 ADC data item stats

Attachment 31 DSSSD bias & leakage current

Conclusion - no change cf. 16.40 test



19.10 Re-order power cabling at AIDA FEE64 PSUs from ...

Top    1-3 2-4 9-10 11-12
Bottom 5-7 6-8 13-14 15-16

to ...

Top    1-2 3-4 9-10 11-12
Bottom 5-6 7-8 13-14 15-16


ASIC check all FEE64s

Attachments 32 & 33 1.8.W spectra 20us FSR

Attachment 34 ADC data item stats

Attachment 35 DSSSD bias & leakage current

Conclusion - all p+n FEE64 rates c. 100k, or less, n+n FEE64 rates c. 400k

FEE64 adaptor PCB config

LK2-LK3-LK4 fitted for all bottom p+n FEE64s (aida11, 3, 12, 15, 7, 16)

LK1 fitted n+n FEE64s aida06 & aida02 *only*
Attachment 1: Screenshot_from_2022-04-03_11-54-21.png
Screenshot_from_2022-04-03_11-54-21.png
Attachment 2: Screenshot_from_2022-04-03_11-54-04.png
Screenshot_from_2022-04-03_11-54-04.png
Attachment 3: Screenshot_from_2022-04-03_11-53-15.png
Screenshot_from_2022-04-03_11-53-15.png
Attachment 4: Screenshot_from_2022-04-03_13-07-21.png
Screenshot_from_2022-04-03_13-07-21.png
Attachment 5: Screenshot_from_2022-04-03_13-06-48.png
Screenshot_from_2022-04-03_13-06-48.png
Attachment 6: Screenshot_from_2022-04-03_13-06-34.png
Screenshot_from_2022-04-03_13-06-34.png
Attachment 7: Screenshot_from_2022-04-03_13-05-24.png
Screenshot_from_2022-04-03_13-05-24.png
Attachment 8: Screenshot_from_2022-04-03_13-21-31.png
Screenshot_from_2022-04-03_13-21-31.png
Attachment 9: Screenshot_from_2022-04-03_13-21-07.png
Screenshot_from_2022-04-03_13-21-07.png
Attachment 10: Screenshot_from_2022-04-03_13-20-43.png
Screenshot_from_2022-04-03_13-20-43.png
Attachment 11: Screenshot_from_2022-04-03_13-20-25.png
Screenshot_from_2022-04-03_13-20-25.png
Attachment 12: Screenshot_from_2022-04-03_13-32-24.png
Screenshot_from_2022-04-03_13-32-24.png
Attachment 13: Screenshot_from_2022-04-03_13-31-50.png
Screenshot_from_2022-04-03_13-31-50.png
Attachment 14: Screenshot_from_2022-04-03_13-31-32.png
Screenshot_from_2022-04-03_13-31-32.png
Attachment 15: Screenshot_from_2022-04-03_13-31-06.png
Screenshot_from_2022-04-03_13-31-06.png
Attachment 16: Screenshot_from_2022-04-03_15-52-42.png
Screenshot_from_2022-04-03_15-52-42.png
Attachment 17: Screenshot_from_2022-04-03_15-51-49.png
Screenshot_from_2022-04-03_15-51-49.png
Attachment 18: Screenshot_from_2022-04-03_15-51-34.png
Screenshot_from_2022-04-03_15-51-34.png
Attachment 19: Screenshot_from_2022-04-03_15-50-53.png
Screenshot_from_2022-04-03_15-50-53.png
Attachment 20: Screenshot_from_2022-04-03_16-40-15.png
Screenshot_from_2022-04-03_16-40-15.png
Attachment 21: Screenshot_from_2022-04-03_16-39-43.png
Screenshot_from_2022-04-03_16-39-43.png
Attachment 22: Screenshot_from_2022-04-03_16-39-34.png
Screenshot_from_2022-04-03_16-39-34.png
Attachment 23: Screenshot_from_2022-04-03_16-39-22.png
Screenshot_from_2022-04-03_16-39-22.png
Attachment 24: Screenshot_from_2022-04-03_16-59-59.png
Screenshot_from_2022-04-03_16-59-59.png
Attachment 25: Screenshot_from_2022-04-03_16-59-33.png
Screenshot_from_2022-04-03_16-59-33.png
Attachment 26: Screenshot_from_2022-04-03_16-59-10.png
Screenshot_from_2022-04-03_16-59-10.png
Attachment 27: Screenshot_from_2022-04-03_16-59-07.png
Screenshot_from_2022-04-03_16-59-07.png
Attachment 28: Screenshot_from_2022-04-03_18-01-39.png
Screenshot_from_2022-04-03_18-01-39.png
Attachment 29: Screenshot_from_2022-04-03_18-00-56.png
Screenshot_from_2022-04-03_18-00-56.png
Attachment 30: Screenshot_from_2022-04-03_18-00-43.png
Screenshot_from_2022-04-03_18-00-43.png
Attachment 31: Screenshot_from_2022-04-03_18-00-40.png
Screenshot_from_2022-04-03_18-00-40.png
Attachment 32: Screenshot_from_2022-04-03_19-10-37.png
Screenshot_from_2022-04-03_19-10-37.png
Attachment 33: Screenshot_from_2022-04-03_19-10-11.png
Screenshot_from_2022-04-03_19-10-11.png
Attachment 34: Screenshot_from_2022-04-03_19-10-00.png
Screenshot_from_2022-04-03_19-10-00.png
Attachment 35: Screenshot_from_2022-04-03_19-09-36.png
Screenshot_from_2022-04-03_19-09-36.png
  407   Sat Apr 2 10:42:49 2022 TDSaturday 2 April
11.10 Found aida08 not running - merger/TapeServer stalled

Restart required power cycle of all FEE64s
*all* ADC calibrations failed during system wide checks it was necessary to use FADC align &amp; control (act on all FEE64 modules selected) to calibrate - first pass 
12/16 calibrated, successive individual re-calibrations calibrated remaining FEE64s

ASIC settings 2021Apr29-13-16-00
 slow comparator 0x2f

BNC PB-5
amplitude 1.0V
attenuation x1
polarity +
tail pulse
tau_d 1ms
rate 25Hz

Attachments 1-4 1.8.W n+n spectra 20us, 200us, 2ms, 20ms FSR
Attachments 5-8 1.8.W p+n spectra 20ms, 2ms, 200us, 20us FSR

Attachment 9 per FEE64 1.8.L spectra n+n strips
 aida04 pulser peak width 387 ch FWHM
Attachment 10 per FEE64 1.8.L spectra p+n strips
 aida10 pulser peak width 93 ch FWHM

Attachment 11 per FEE64 Rate spectra

Attachments 12-15 all system wide checks OK
Attachment 16 ADC data item stats
Attachment 17 FEE64 temps OK
Attachment 18 DSSSD bias & leakage currents OK

15.45 check of ASIC settings cf. https://www2.ph.ed.ac.uk/~td/AIDA/Information/Default%20AIDA%20ASIC%20Parameters.txt

Differences from default AIDA ASIC parameters
--------------------------------------------- 

                                            Input Polarity
Offset Title                                Positive   Negative            Notes

12     slow comparator threshold            0x2f->0xa                     100keV nominal
 
14     fast comparator threshold HEC        0xf->0x02                     200MeV nominal
15     fast comparator threshold LEC/MEC    0xf->0xff                     > LEC/MEC nominal FSR

18     preAmp reference                     0xb2       0x20->0x30

21     Ibias LF feedback                    0x08       0xb->0xf

26     diode link threshold                 0xca->0xbf 0x11->0x23         decrease switch threshold

(front) p+n junction strips - positive input polarity
(back) n+n ohmic strips - negative input polarity

All looks OK - variations from default settings are those used at RIKEN

16:45 FEE64 adaptor PCB config

    LK1 LK2 LK3 LK4
p+n n/c o   *   o
n+n o   o   n/c o

o   = jumper fitted
n/c = jumper not fitted

*   = top, middle FEE64s - n+n bias ring ground for Si wafer #2
      bottom, middle FEE64s - DSSSD PCB gnd

DSSSD#1 & #2 bias via standard coaxial cable to top p+n FEE64 adaptor PCBs

17.05 aida09 or aida13 cable braid not secured to ground - cable tag dia too large - now sandwiched between correctly sized cable tags 
      no obvious difference to adc data item rates


17.20 LK1 removed for all n+n PCBs
      attachment 19 - adc data item stats

18.55 LK2/LK4 removed for all n+n PCBs
      attachment 20 - adc data item stats

19.10 power cycle FEE64s
      attachment 21 - adc data item stats
Attachment 1: Screenshot_from_2022-04-02_11-50-09.png
Screenshot_from_2022-04-02_11-50-09.png
Attachment 2: Screenshot_from_2022-04-02_11-49-44.png
Screenshot_from_2022-04-02_11-49-44.png
Attachment 3: Screenshot_from_2022-04-02_11-49-20.png
Screenshot_from_2022-04-02_11-49-20.png
Attachment 4: Screenshot_from_2022-04-02_11-49-02.png
Screenshot_from_2022-04-02_11-49-02.png
Attachment 5: Screenshot_from_2022-04-02_11-48-38.png
Screenshot_from_2022-04-02_11-48-38.png
Attachment 6: Screenshot_from_2022-04-02_11-47-54.png
Screenshot_from_2022-04-02_11-47-54.png
Attachment 7: Screenshot_from_2022-04-02_11-46-51.png
Screenshot_from_2022-04-02_11-46-51.png
Attachment 8: Screenshot_from_2022-04-02_11-46-11.png
Screenshot_from_2022-04-02_11-46-11.png
Attachment 9: Screenshot_from_2022-04-02_11-44-14.png
Screenshot_from_2022-04-02_11-44-14.png
Attachment 10: Screenshot_from_2022-04-02_11-41-09.png
Screenshot_from_2022-04-02_11-41-09.png
Attachment 11: Screenshot_from_2022-04-02_11-39-57.png
Screenshot_from_2022-04-02_11-39-57.png
Attachment 12: Screenshot_from_2022-04-02_11-39-02.png
Screenshot_from_2022-04-02_11-39-02.png
Attachment 13: Screenshot_from_2022-04-02_11-38-55.png
Screenshot_from_2022-04-02_11-38-55.png
Attachment 14: Screenshot_from_2022-04-02_11-38-47.png
Screenshot_from_2022-04-02_11-38-47.png
Attachment 15: Screenshot_from_2022-04-02_11-38-35.png
Screenshot_from_2022-04-02_11-38-35.png
Attachment 16: Screenshot_from_2022-04-02_11-38-19.png
Screenshot_from_2022-04-02_11-38-19.png
Attachment 17: Screenshot_from_2022-04-02_11-37-51.png
Screenshot_from_2022-04-02_11-37-51.png
Attachment 18: Screenshot_from_2022-04-02_11-37-30.png
Screenshot_from_2022-04-02_11-37-30.png
Attachment 19: Screenshot_from_2022-04-02_17-20-12.png
Screenshot_from_2022-04-02_17-20-12.png
Attachment 20: Screenshot_from_2022-04-02_18-54-11.png
Screenshot_from_2022-04-02_18-54-11.png
Attachment 21: Screenshot_from_2022-04-02_19-07-42.png
Screenshot_from_2022-04-02_19-07-42.png
  406   Wed Mar 9 10:15:56 2022 NH TDWednesday 9 March - noise tests contd.
11.15 - S4 is currently closed no access

If access possible (should be) plan is

Re-attach ribbon cable braids
Re-attach DSSSD middle wafer rear g/r (LK3)
Re-attach DSSSD ground (LK2?)

See difference in rates, expectation is braid should restore most p+n rates

Once status quo ante achieved explore ASIC settings to identify preamp feedback and shaping time options to stabilise preamp output and optimise signal:noise.

aida12 Peak width =    84.73
aida02 Peak width =   526.68
Attachment 1 - Stats
Attachment 2 - Per FEE Rates
Attachments 3,4 - Waveforms
Attachments 5,6 - Pulser Peaks
Attachment 7 - HV Bias. Leakage current recovered and does not seem to be ramping up

16.00 - Attach LK3 jumpers to aidas 1, 3, 5, 7 (DSSD ground and middle rear g/r)

Attachment 8 - Stats
Attachment 9 - Per FEE Rates
Attachment 10, 11 - Waveforms
Attachment 12, 13 - Pulser Peaks

aida12 Peak width =    78.51
aida02 Peak width =   480.90

No major change, FEES 1-8 are all noisy and FEES 9-16 are quiet
aida01, aida05 waveforms very noticably bad (100 kHz), aida03, aida07 waveforms seem OK


16.45 - Attach LK2, LK4 jumpers to all FEEs (field plane grounding)
No significant changes

Attachment 14 - Stats
Attachment 15 - Per FEE Rates
Attachment 16, 17 - Pulser Peaks
Attachment 18, 19 - Waveforms

aida12 Peak width =    71.28
aida02 Peak width =   458.18

Remove LK3 jumpers 

aida12 Peak width =    71.43 
aida02 Peak width =   496.05


Attachment 20 - Stats
Attachment 21 - Per FEE Rates
Attachment 22, 23 - Pulser Peaks
Attachment 24, 25 - Waveforms

Re-add jumpers - attachments 26-28

aida12 Peak width =    87.42
aida02 Peak width =   572.24

17.45 Directly bias from the p+n FEEs using SHV->BNC->Lemo - attachments 29-33

aida12 Peak width =    87.87
aida02 Peak width =   504.41

The correlation between high 'ADC data item' rate and the noise observed on the corresponding 1.8.W spectrum is not clear *except* for aida01 & aida05 - see also attachments 34-35

ADC data item stats - attachment 36

DSSSD bias & leakage current OK - attachment 37

FEE64 temperatures OK - attachment 38

all system wide checks OK *except aida11 clock fail status 6
Attachment 1: 2022-03-09_15-37-08_stats.png
2022-03-09_15-37-08_stats.png
Attachment 2: 2022-03-09_15-37-33_rates.png
2022-03-09_15-37-33_rates.png
Attachment 3: 2022-03-09_15-38-18_pn_waves.png
2022-03-09_15-38-18_pn_waves.png
Attachment 4: 022-03-09_15-38-59_nn_waves.png
022-03-09_15-38-59_nn_waves.png
Attachment 5: 2022-03-09_15-40-23_pn_widths.png
2022-03-09_15-40-23_pn_widths.png
Attachment 6: 2022-03-09_15-41-26_nn_widths.png
2022-03-09_15-41-26_nn_widths.png
Attachment 7: 2022-03-09_15-42-55_hv_bias.png
2022-03-09_15-42-55_hv_bias.png
Attachment 8: 2022-03-09_16-03-05_stats.png
2022-03-09_16-03-05_stats.png
Attachment 9: 2022-03-09_16-06-30_rates.png
2022-03-09_16-06-30_rates.png
Attachment 10: 2022-03-09_16-07-43_pn_waves.png
2022-03-09_16-07-43_pn_waves.png
Attachment 11: 2022-03-09_16-08-12_nn_waves.png
2022-03-09_16-08-12_nn_waves.png
Attachment 12: 2022-03-09_16-10-10_pn_widths.png
2022-03-09_16-10-10_pn_widths.png
Attachment 13: 2022-03-09_16-11-14_nn_widths.png
2022-03-09_16-11-14_nn_widths.png
Attachment 14: 2022-03-09_16-40-33_stats.png
2022-03-09_16-40-33_stats.png
Attachment 15: 2022-03-09_16-42-44_rates.png
2022-03-09_16-42-44_rates.png
Attachment 16: 2022-03-09_16-43-46_pn_widths.png
2022-03-09_16-43-46_pn_widths.png
Attachment 17: 2022-03-09_16-44-46_nn_widths.png
2022-03-09_16-44-46_nn_widths.png
Attachment 18: 2022-03-09_16-45-39_pn_waves.png
2022-03-09_16-45-39_pn_waves.png
Attachment 19: 2022-03-09_16-46-08_nn_waves.png
2022-03-09_16-46-08_nn_waves.png
Attachment 20: 2022-03-09_17-03-33_stats.png
2022-03-09_17-03-33_stats.png
Attachment 21: 2022-03-09_17-04-22_rates.png
2022-03-09_17-04-22_rates.png
Attachment 22: 2022-03-09_17-07-44_nn_widths.png
2022-03-09_17-07-44_nn_widths.png
Attachment 23: 2022-03-09_17-06-40_pn_widths.png
2022-03-09_17-06-40_pn_widths.png
Attachment 24: 2022-03-09_17-05-41_nn_waves.png
2022-03-09_17-05-41_nn_waves.png
Attachment 25: 2022-03-09_17-05-11_pn_waves.png
2022-03-09_17-05-11_pn_waves.png
Attachment 26: 2022-03-09_17-26-03_nn_widths.png
2022-03-09_17-26-03_nn_widths.png
Attachment 27: 2022-03-09_17-24-58_pn_widths.png
2022-03-09_17-24-58_pn_widths.png
Attachment 28: 2022-03-09_17-23-57_stats.png
2022-03-09_17-23-57_stats.png
Attachment 29: 2022-03-09_17-43-33nn_widths.png
2022-03-09_17-43-33nn_widths.png
Attachment 30: 2022-03-09_17-42-29_pn_widths.png
2022-03-09_17-42-29_pn_widths.png
Attachment 31: 2022-03-09_17-41-23_stats.png
2022-03-09_17-41-23_stats.png
Attachment 32: 2022-03-09_17-45-55_nn_waves.png
2022-03-09_17-45-55_nn_waves.png
Attachment 33: 2022-03-09_17-45-27_pn_waves.png
2022-03-09_17-45-27_pn_waves.png
Attachment 34: Screenshot_from_2022-03-09_18-12-13.png
Screenshot_from_2022-03-09_18-12-13.png
Attachment 35: Screenshot_from_2022-03-09_18-11-32.png
Screenshot_from_2022-03-09_18-11-32.png
Attachment 36: Screenshot_from_2022-03-09_18-27-23.png
Screenshot_from_2022-03-09_18-27-23.png
Attachment 37: Screenshot_from_2022-03-09_18-32-13.png
Screenshot_from_2022-03-09_18-32-13.png
Attachment 38: Screenshot_from_2022-03-09_18-35-24.png
Screenshot_from_2022-03-09_18-35-24.png
  405   Tue Mar 8 10:26:56 2022 TD NHTuesday 8 March 2022
11.25 System wide checks OK except aida06 clock fail status 6 - attachments 1-13

      ADC data item stats - attachment 14

      FEE64 temperatures OK - attachment 15

      DSSSD bias & leakage currents - attachment 16
  
      DSSSD#1 OK. DSSSD#2 leakage current has increased from c.5uA to c. 11uA overnight.


11.43 per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - attachments 17-24

      per FEE64 1.8.L spectra - attachments 25-26
          aida12 1.8.L pulser peak width 143 ch FWHM
          aida02 1.8.L pulser peak width 439 ch FWHM

      Per FEE64 rate spectra - attachment 27

12.00 Cycled DSSSD#2 bias from -100V to 0 to -100V leakage current -7.7uA but ramping slowly (c. 5nA/10s ... ?)

13.26 DSSSD#2 bias -100V leakage current -8.54uA

15.46 - Noticed that DSSSD#1 braid no longer has filter, DSSSD#1 core still does. DSSSD#2 is unfiltered on both.

      - Swap FEE64 power connectors #3 with #4, and #6 with #7, this means both n+n FEE64s are on the same paired channel

      No significant change to stats

      - Notice braid was only connected to FEEs 4 and 8, which meant beam-left bias guard ring 0v was floating. Use T to bring bias braid over to FEES 2 and 6 as well.
        Middle bias guard ring 0v is still floating

      No significant change to stats, amplitude of 100 kHz waveform in n+n waves seems attenuated though

     aida12 1.8.L pulser peak width 76 ch FWHM
     aida02 1.8.L pulser peak width 438 ch FWHM
 
     Attachment 28 - HV leakage current
     Attachment 29 - FEE ADC stats
     Attachment 30 - p+n side waveforms
     Attachment 31 - n+n side waveforms
     Attachment 32 - p+n pulser peak widths
     Attachment 33 - n+n pulser peak widths

15.55 - Reattach filter to DSSSD#1 braid

     No major change, possible improvement in waveforms 
     Attachment 34 - Stats
     Attachment 35,36 - waveforms

16.46 - Reattach LK1 jumper to FEES 2, 4, 6, 8

     Waveforms look OK, p+n rate slightly worse?
     Attachment 37 - Stats
     Attachment 38, 39 - waveforms
     Attachment 40 - Per FEE Rates
Attachment 1: Screenshot_from_2022-03-08_11-25-00.png
Screenshot_from_2022-03-08_11-25-00.png
Attachment 2: Screenshot_from_2022-03-08_11-24-55.png
Screenshot_from_2022-03-08_11-24-55.png
Attachment 3: Screenshot_from_2022-03-08_11-24-48.png
Screenshot_from_2022-03-08_11-24-48.png
Attachment 4: Screenshot_from_2022-03-08_11-24-42.png
Screenshot_from_2022-03-08_11-24-42.png
Attachment 5: Screenshot_from_2022-03-08_11-24-35.png
Screenshot_from_2022-03-08_11-24-35.png
Attachment 6: Screenshot_from_2022-03-08_11-24-30.png
Screenshot_from_2022-03-08_11-24-30.png
Attachment 7: Screenshot_from_2022-03-08_11-24-23.png
Screenshot_from_2022-03-08_11-24-23.png
Attachment 8: Screenshot_from_2022-03-08_11-24-15.png
Screenshot_from_2022-03-08_11-24-15.png
Attachment 9: Screenshot_from_2022-03-08_11-23-09.png
Screenshot_from_2022-03-08_11-23-09.png
Attachment 10: Screenshot_from_2022-03-08_11-22-45.png
Screenshot_from_2022-03-08_11-22-45.png
Attachment 11: Screenshot_from_2022-03-08_11-22-36.png
Screenshot_from_2022-03-08_11-22-36.png
Attachment 12: Screenshot_from_2022-03-08_11-21-52.png
Screenshot_from_2022-03-08_11-21-52.png
Attachment 13: Screenshot_from_2022-03-08_11-21-41.png
Screenshot_from_2022-03-08_11-21-41.png
Attachment 14: Screenshot_from_2022-03-08_11-21-30.png
Screenshot_from_2022-03-08_11-21-30.png
Attachment 15: Screenshot_from_2022-03-08_11-21-20.png
Screenshot_from_2022-03-08_11-21-20.png
Attachment 16: Screenshot_from_2022-03-08_11-21-02.png
Screenshot_from_2022-03-08_11-21-02.png
Attachment 17: Screenshot_from_2022-03-08_11-40-39.png
Screenshot_from_2022-03-08_11-40-39.png
Attachment 18: Screenshot_from_2022-03-08_11-40-03.png
Screenshot_from_2022-03-08_11-40-03.png
Attachment 19: Screenshot_from_2022-03-08_11-39-35.png
Screenshot_from_2022-03-08_11-39-35.png
Attachment 20: Screenshot_from_2022-03-08_11-39-14.png
Screenshot_from_2022-03-08_11-39-14.png
Attachment 21: Screenshot_from_2022-03-08_11-38-07.png
Screenshot_from_2022-03-08_11-38-07.png
Attachment 22: Screenshot_from_2022-03-08_11-37-33.png
Screenshot_from_2022-03-08_11-37-33.png
Attachment 23: Screenshot_from_2022-03-08_11-37-14.png
Screenshot_from_2022-03-08_11-37-14.png
Attachment 24: Screenshot_from_2022-03-08_11-35-51.png
Screenshot_from_2022-03-08_11-35-51.png
Attachment 25: Screenshot_from_2022-03-08_11-35-16.png
Screenshot_from_2022-03-08_11-35-16.png
Attachment 26: Screenshot_from_2022-03-08_11-32-45.png
Screenshot_from_2022-03-08_11-32-45.png
Attachment 27: Screenshot_from_2022-03-08_11-30-24.png
Screenshot_from_2022-03-08_11-30-24.png
Attachment 28: 2022-03-08_15-48-40_HV.png
2022-03-08_15-48-40_HV.png
Attachment 29: 2022-03-08_15-49-03_stats.png
2022-03-08_15-49-03_stats.png
Attachment 30: 2022-03-08_15-53-42_pn_waves.png
2022-03-08_15-53-42_pn_waves.png
Attachment 31: 2022-03-08_15-50-12_nn_waves.png
2022-03-08_15-50-12_nn_waves.png
Attachment 32: 2022-03-08_15-54-40_pn_widths.png
2022-03-08_15-54-40_pn_widths.png
Attachment 33: 2022-03-08_15-57-00_nn_widths.png
2022-03-08_15-57-00_nn_widths.png
Attachment 34: 2022-03-08_16-13-21_stats.png
2022-03-08_16-13-21_stats.png
Attachment 35: 2022-03-08_16-15-35_pn_waves.png
2022-03-08_16-15-35_pn_waves.png
Attachment 36: 2022-03-08_16-14-58_nn_waves.png
2022-03-08_16-14-58_nn_waves.png
Attachment 37: Screenshot_from_2022-03-08_16-39-44.png
Screenshot_from_2022-03-08_16-39-44.png
Attachment 38: Screenshot_from_2022-03-08_16-40-15.png
Screenshot_from_2022-03-08_16-40-15.png
Attachment 39: Screenshot_from_2022-03-08_16-41-08.png
Screenshot_from_2022-03-08_16-41-08.png
Attachment 40: Screenshot_from_2022-03-08_16-41-57.png
Screenshot_from_2022-03-08_16-41-57.png
  404   Mon Mar 7 12:20:26 2022 NH, TDAIDA Grounding Checks Mon 7 Mar
Remove all DSSD ribbon cable braid ground from FEE64s to check n+n ground loops

Removing all on n+n side only and test resistance to ground: 4, 6 and 8 show continuity to ground, 2 shows no connection (braid ring terminal to copper bar)

Remove all p+n braids as wll and check:
- See no continuity to ground (good)
- Snout has no continuity to ground (good)
- n+n side fees 4 and 8 to seem to have continuity to snout though (bad)
- p+n checked and don't seem to have continuity to gnd (except when braid was touching a LEMO barrel)
- n+n aida06 maybe has high resistance connection to ground (bad) or (about 0.7 V drop according to RS multimeter)

Bias+Power up and check noise
aida09 Peak width = 86.03+/- 1.721
aida02 Peak width = 509.48+/- 11.1812

Low frequency noise on wavesforms corresponds to ca 100 kHz - switching power supply?

Attachment 6 - ADC data item stats
all p+n/n+n cable braids disconnected
all FEE64 adaptor PCBS grounded to Copper busbar OK

Remove guard ring jumper on upper-middle p+n FEE (rear guard ring of middle DSSSD)
Attachment 7 - ADC data item states
No real change to rates

Remove field plate jumpers on all p+n FEEs
Attachment 8 - ADC data item states
No real change to rates

Remove bias LK1 jumpers on n+n FEES:
Generally worse across the p+n side, no real change to n+n side

Attachment 9 - Stats with LK1 removed:
aida09 Peak Width = 181.77. p+n see Attachment 10
aida02 Peak width = 505.23. n+n see Attachment 11

Attachment 12 - p+n waveforms
Attachment 13 - n+n waveforms

High frequency component reduced but 100 kHz low frequency component stronger

Remove LK3 from aida03,aida07 (no jumpers at all on any FEE) - DSSSDs floating
No significant change, see attachment 14 stats
Attachment 1: 2022-03-07_14-00_Stats.png
2022-03-07_14-00_Stats.png
Attachment 2: 2022-03-07_14-03-03_pn_widths.png
2022-03-07_14-03-03_pn_widths.png
Attachment 3: 2022-03-07_14-04-13_nn_widths.png
2022-03-07_14-04-13_nn_widths.png
Attachment 4: 022-03-07_14-05-12_pn_waves.png
022-03-07_14-05-12_pn_waves.png
Attachment 5: 2022-03-07_14-05-49_nn_waves.png
2022-03-07_14-05-49_nn_waves.png
Attachment 6: Screenshot_from_2022-03-07_15-28-41.png
Screenshot_from_2022-03-07_15-28-41.png
Attachment 7: 2022-03-07_15-48-02_stats_no_middle_rear_GR.png
2022-03-07_15-48-02_stats_no_middle_rear_GR.png
Attachment 8: 2022-03-07_16-08-11_stats_no_fieldplane_ground.png
2022-03-07_16-08-11_stats_no_fieldplane_ground.png
Attachment 9: 2022-03-07_16-22-00_stats_no_LK1.png
2022-03-07_16-22-00_stats_no_LK1.png
Attachment 10: 2022-03-07_16-23-50_pn_widths_no_lk1.png
2022-03-07_16-23-50_pn_widths_no_lk1.png
Attachment 11: 2022-03-07_16-24-55_nn_widths_no_lk1.png
2022-03-07_16-24-55_nn_widths_no_lk1.png
Attachment 12: 2022-03-07_16-25-50_pn_waves_no_lk1.png
2022-03-07_16-25-50_pn_waves_no_lk1.png
Attachment 13: 2022-03-07_16-27-09_nn_waves_no_lk1.png
2022-03-07_16-27-09_nn_waves_no_lk1.png
Attachment 14: 2022-03-07_17-12-34_no_detector_gnd__stats.png
2022-03-07_17-12-34_no_detector_gnd__stats.png
  403   Sat Mar 5 10:43:45 2022 OH, NHGrounding and noise tests 4th-5th March Summary

Tests performed

  • After leaving overnight noticed the pulser was not fully inserted into aida09
    • Little change in rates observed in 9 but otherwise nothing else
  • Connected bias of aida02 (n+n ground) to aida06 (n+n ground) and likewise 4-8
    • No change observed in rates
  • Also tested covering the snout in light tight material
    • No Changes
  • Unplugged the system monitor from 6 and 8, no change
  • Connected the snout to ground with copper tape
    • Rates became very unpredictable in the FEEs unsure if this was snout grounding related though
    • Removed the ground and powercycled and rates restored to where previously
  • Changed the bias supply jumper inside to have the bias line connected to the rack ground
    • At the same time noticed that the ground cable connecting the adaptor board of aida06 to the FEE had become disconnected from the ring terminal
      • Recrimped and inserted again
    • Saw an improvment in the rates of the p+n strips and also the n+n strips - attachment 1
    • Pulser widths however larger than yesterday

p+n width

FEE Width
9 118
1 88
10 75
11 77
3 67
12 66
13 96
5 89
14 107
15 93
7 118
16 87

n+n width

  • Connected up entire pulser circuit to n+n FEEs
FEE Width
2 428
4 N/A
6 428
8 343

Switching bias supply back

  • because changes were made to the n+n ground, and the pulser circuit it was decided to go back and check the previous bias configuration with the module set in a floating configuration
  • DAQ was then left overnight as had to leave to catch the bus
  • Rates are observed to be back at their higher values - attachment 2
  • n+n 1.8.L spectra - attachment 3
  • p+n 1.8.L spectra - attachment 4

p+n widths

FEE Wdith
9 130
1 128
10 75
11 78
3 71
12 71
13 96
5 85
14 110
15 99
7 123 Dbl
16 92

n+n widths

FEE width
2 56
4 983
6 523
8 358
  • The results show no improvment to slightly worse performance so will go back to having the bias module grounded internally and check the previous conclusions hold before making further changes

Return to grounded bias module

Widths

p+n

FEE Width
9 106
1 79
10 72
11 78
3 70
12 73
13 102
5 112
14 105
15 110
7 131
16 104

n+n

FEE Width
2 -
4 298
6- -
8 278
  • Don't see a pulser peak in 4 or 2
  • Will replace cable between 4 and 2 and 2 and 6
  • Statistics at 0x64 with pulser on - atachment 5
  • Now see peaks - attachment 6
  • Statistics at 0xa - attachment 7
FEE Width
2 284
4 1089 (Triple)
6 371
8 318

With the grounded bias in place in the module move the braid cable to the frame rather than being connected to bias line

  • Still observe a triple peak in 4 and have worse perfromance in 2 and no improvement in p+n
  • 0x64 stats - attachment 7
  • 0xa statistics - attachment 8
  • p+n 1.8.L - attachment 9
  • p+n waveform - attachment 10
  • n+n waveform - attachment 11
    • Same high frequency component in both
    • 35 channel spacing corresponds to a frequency of ~1.4MHz
FEE p+n Width
9 109
1 118
10 78
11 80
3 70
12 73
13 92
5 85
14 99
15 100
7 123
16 92
FEE n+n Width
2 531
4 1185 (Triple)
6 536
8 376

Next tests

  • Returned braid to FEE 4 and 8
    • Recovered earlier rates and performance
  • Re-added the ground cable to the snout
    • Much worse noise performance across all FEEs
    • p+n rates in 200k
  • Removed the ground from the snout
    • Got really good rates originally - attachment 12
    • Noticed 1 and 5 failed ADC calibration
    • Calibrated them and there rates increased massively - attachment 13
  • Notice that the pulser peak in FEE2 and 6 keeps dropping out
    • As the cables have been changed multiple times thought it could be that an adaptor board was broken
    • 4 seemed the most likely suspect with its high rates and triple peaks
    • Replaced the adaptor board with one missing an outer ERNI pin (PIN doesn't interface with the FEE so didn't see any issues)
    • Pulser peaks were still muissing
    • Issue could be then with adaptor board 2 possibly

Conclusions

  • Rates on the FEEs are in a better place now than they were at the start of the week
    • p+n in particular have shown a large amount of improvement
  • There are still however issues with the n+n FEEs
  • I think there is likely an issue with one of the adaptor boards
    • Possible aida02 at the pulser connection point
    • Could explain the triple peaking in aida04 and the pulser issues with aida02 and aida06 dropping out.
    • Would recomend swapping out aida02 as a test next week
  • Other possible issues coiuld be the clingfilm around the cables has torn and we are grounding the cables to the inside of the snout
    • Nic would like to use something other than clingfilm in future
    • One suggestion would be mylar which could be held in place with strips of double sided tape
    • This would make the cables less likely to stick to each other which is an issue with the tightness of the snout
  • Could also try removing LK3 from FEE3 and 7 leaving the PCB ground floating
  • I am at a slight loss of what else to try

 

 

  • I have left the daq running at 0x64 overnight to check for alphas
  • Histograms and statistics zeroed
Attachment 1: 220304_1704_groundedbias_stats.png
220304_1704_groundedbias_stats.png
Attachment 2: 220305_1154_Stats.png
220305_1154_Stats.png
Attachment 3: 220305_1157_nn_18L.png
220305_1157_nn_18L.png
Attachment 4: 220305_1200_pn_18L.png
220305_1200_pn_18L.png
Attachment 5: 220305_1336_0x64Stats.png
220305_1336_0x64Stats.png
Attachment 6: 220305_1341_nn_18L.png
220305_1341_nn_18L.png
Attachment 7: 220305_1342_Stats.png
220305_1342_Stats.png
Attachment 8: 220305_1405_0x64_stats.png
220305_1405_0x64_stats.png
Attachment 9: 220305_1421_pn_18L.png
220305_1421_pn_18L.png
Attachment 10: 220305_1424_pn_waveform.png
220305_1424_pn_waveform.png
Attachment 11: 220305_1429_nn_waveform.png
220305_1429_nn_waveform.png
Attachment 12: 220305_1518_stats.png
220305_1518_stats.png
Attachment 13: 220305_1519_waveform_enabled.png
220305_1519_waveform_enabled.png
  402   Fri Mar 4 08:40:36 2022 OH, NHAIDA Noise and grounding tests 3rd March - Results

Grounding implemented

  • Each FEE given direct connection to heavy duty earth bar via adaptor board lemo ground
  • Earth bars are then connected to platform ground on AIDA frame
  • Photo of grounding - attachment 1
  • Improvements not as large as expected from previous experience though
    • Particularly in the n+n side where widths still appear awful
  • While implementing the grounding it was noticed the ground lemo socket on FEE7 was damaged and no longer gripped onto a lemo cable
    • It was noted that it would need changing at some point but continued with it for the time being
  • Jumper configuration was n+n LK1-4 all on, p+n LK2-4 all on

p+n widths

FEE Width
9 147
1 116
10 92
11 77
3 128
12 72
13 75
5 59
14 75
15 80
7 150
16 75
  • Compared to the width with no grounding the FEEs for the first DSSD (First 6) range from no change to slightly worse
  • 1.8.L spectra - attachment 3

n+n widths

  • Widths still very large
  • Width for 8 is around 500
  • 1.8.L attachment 2

Additional info

  • Statistics - attachment 4

Jumper optimisation

  • Went to our previously found optimal jumper configuration for the trips
  • n+n LK1 only
  • p+n LK 2 and 4 on all FEE
  • p+n LK3 on non isolated cables (PCB ground on lower middle cable and n+n side bias on top middle cable)
  • While changing the jumpers we also swapped out the damaged adaptor board on aida07 for a new one
  • Saw an improvement for most FEEs
FEE Width
9 134
1 83
10 80
11 68
3 60
12 60
13 71
5 60
14 75
15 82
7 97
16 77
  • Unfortunately saw no improvement for the n+n
  • We event tested having the pulser only going into a single FEE (aida08)
  • Still have a width of 622 for that FEE
  • To compare FEEs we have layout 1 - attachment 5
  • Layouot 1 on a common timescale - attachment 6
    • Note that the rate is typically dominated by a single strip or a couple of strips within a FEE
    • Many channels are just showing the pulser rate 25Hz
  • p+n  1.8.L attachment 7
  • p+n waveforms attachment 8
  • n+n waveforms attachment 9
  • Testing the noise on the n+n strips we observe noise with slow comparator set to 200 (2MeV)
  • Meanwhile the p+n strips are silent at 0x64 for the most part.
  • DAQ left running overnight with 0x64 on ASIC settings to see if we observe any alphas

Plan for the day

  • Test daisy chaining the n+n across to the other n+n strips
    • Currently the n+n braid is only plugged into either aida06 or aida08 (depending on DSSD)
    • All of the jumpers are however grounded
    • Will still test
    • May investigate also trying this on LK3 on adaptors for aida01 and aida03
  • Observe the output of the new sum inverter on the scope to check for variation in amplitude
Attachment 1: PXL_20220303_104253926.jpg
PXL_20220303_104253926.jpg
Attachment 2: 220303_1310_nn_18l.png
220303_1310_nn_18l.png
Attachment 3: 220303_1313_pn_18L.png
220303_1313_pn_18L.png
Attachment 4: 220303_1313_stats.png
220303_1313_stats.png
Attachment 5: 220303_1438_layout1.png
220303_1438_layout1.png
Attachment 6: 220303_1439_layout1common.png
220303_1439_layout1common.png
Attachment 7: 220303_1446_pn_18L.png
220303_1446_pn_18L.png
Attachment 8: 220303_1452_pn_waveform.png
220303_1452_pn_waveform.png
Attachment 9: 220303_1453_nn_waveforms.png
220303_1453_nn_waveforms.png
  401   Thu Mar 3 08:45:03 2022 OH, NHAIDA Noise and grounding tests 3rd March

Pulser spectra

  • Changes to the nn pulser circuit yesterday have improved the peak widths on the p+n side
  • p+n peak widths averact around 100 channels
  • n+n around 1000 at 0xa threshold
  • n+n 1.8.L spectra - attachment 1
  • p+n 1.8.L spectra attachment 2

p+n peak widths

FEE Width
9 105
1 82
10 71
11 79
3 96
12 72
13 191
5 101
14 129
15 118
7 110
16

100

n+n peak widths

FEE Width
2 1200
4 1179
6 -
8 1813

Daily checks

  • Fee temperatures ok - attachment 3
  • Bias and leakage currents ok - attachment 4
  • Stats spectra with current grounding - attachment 5
    • Current grounding is just a cable going between each adaptor board and its respective FEE
    • Interfee grounding done by power supply and also pulser circuit
  • Rates for each of the asics - attachment 6
  • p+n waveforms - attachment 7
  • n+n waveforms - attachment 8
    • Lots of noise evident on them

Plans for the day

  • Connect FEEs up to the copper earth bars installed
  • Powercycle and check noise levels
  • If n+n remains significantly higher investigate the n+n pulser circuit further
  • Investigate the effects of different jumper connetions At present n+n have all LK jumpers on and p+n have lk2-4
Attachment 1: 220303_0927_nn_18L.png
220303_0927_nn_18L.png
Attachment 2: 220303_0928_pn_18L.png
220303_0928_pn_18L.png
Attachment 3: 220303_0932_TEMP.png
220303_0932_TEMP.png
Attachment 4: 220303_0933_Bias.png
220303_0933_Bias.png
Attachment 5: 220303_0933_Stats.png
220303_0933_Stats.png
Attachment 6: 220303_0936_Layout1.png
220303_0936_Layout1.png
Attachment 7: 220303_0941_pn_waveform.png
220303_0941_pn_waveform.png
Attachment 8: 220303_0942_nn_waveform.png
220303_0942_nn_waveform.png
  400   Wed Mar 2 16:00:01 2022 OH, NHNoise tests - 2nd March
An interesting find with the new sum inverter is that with the pulser at default settings the pulser peak on the n+n side is above the range of the ADC.
We have checked signals from the sum inverter and pulser and both are equal in magnitude.
We therefor determine it is over range as the charge is spread over 4 FEEs vs 12 on the p+n side.
This possibly wasn't an issue with the previous inverter if it produced a lower amplitude.

For tests going forward p+n voltage will be 2V. n+n voltage will be 0.5V - Attachment 1
Going to look into a 20dB attenuator

Seems there were issues with the lemo cables in the n+n pulser chain
  399   Wed Mar 2 10:05:13 2022 OHDAQ Status 2nd March 2022
> DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
> Temperature menu is still responsive and all FEEs report a temperature
> Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
> Merger shows no data arriving as expected - Attachment 3
> Merger data buffers for each FEE - attachment 4
> Merger time errors - attachment 5
> 
> DAQ has not yet been restarted in case Patrick wanted to look.

Looking at the first attachment..... aida09 has rebooted. So it is not setup at all.

This has caused the NewMerger to hang .... which surprises me .... as I thought it was designed to spot a bad FEE and continue with the rest.

Yes I will have a look today as I'm in Daresbury.

More later ......
  398   Wed Mar 2 09:08:44 2022 OHDAQ Status 2nd March 2022
DAQ was found yesterday with none of the FEEs reporting any good events on any FEE - attachment 1
Temperature menu is still responsive and all FEEs report a temperature
Looking at the white rabbit timestamps aida09 has a gibberish timestamp - attachment 2
Merger shows no data arriving as expected - Attachment 3
Merger data buffers for each FEE - attachment 4
Merger time errors - attachment 5

DAQ has not yet been restarted in case Patrick wanted to look.
Attachment 1: 220302_1002_goodevents.png
220302_1002_goodevents.png
Attachment 2: 220302_1002_white_rabbit.png
220302_1002_white_rabbit.png
Attachment 3: 220302_1003_merger.png
220302_1003_merger.png
Attachment 4: 220302_1003_merger_data_buffers.png
220302_1003_merger_data_buffers.png
Attachment 5: 220302_1004_merger_merge_time_errors.png
220302_1004_merger_merge_time_errors.png
  397   Tue Mar 1 08:29:19 2022 OH, NHFindings from the upgrade to centos7 from SL6

Required packages not included as default

  • csh (For running MIDAS scripts)
  • xinetd (Runs the rdate server for the FEEs)
  • Telnet (Communication with the FEEs)
  • dhcp

Configuration changes

  • FEEs uses NFS2 which is no longer enabled by default
    • Must edit config in /etc/nfs.conf to uncomment the following lines:
      • [nfsd]
      • vers2=y
  • /etc/xinetd.d/time-stream must be edited by uncommenting
    • disable        = no

  • Many services such as xinetd and dhcp will not automatically start at startup following a restart of the system

    • To enable this must enter `systemctl enable dhcpd` or `systemctl enable xinetd`

    • Likewise can check status with `systemctl status $SERVICE_NAME`

  • Also need to check the firewall status of the ethernet port connected to the FEEs.

    • Easiest solution is to set the port to be trusted

Optional extras

  • For top bar icons similar to SL6 can use the gnome extension "Frippery panel favorites"
    • Then just need to create desktop icons for each of the startup scripts within /usr/share/applications - Attachment 1
    • Can also add screenshot to this - Attachment 2
  • Gnome tweaks also allows the number of workspaces to be changed
  • Also recommend setting workspaces to span displays

 

Attachment 1: TapeServer.desktop
[Desktop Entry]
Name=TapeServer
Exec=/MIDAS/Linux/startup/TapeServer
Terminal=true
Icon=/home/npg/Downloads/midas100.gif
Type=Application
Comment=TapeServer for AIDA

Attachment 2: Screenshot_from_2022-03-01_09-40-18.png
Screenshot_from_2022-03-01_09-40-18.png
  396   Wed Feb 16 10:32:52 2022 PJCSAIDA software changes and some suggestions

The Pi console monitor program now outputs a full date/time to file for each line of report.

The System Wide Checks has been upgraded to include a check of the PLL lock monitor counters. A baseline is taken when the software starts, or at the users command, then subsequent operation of the command compares the current counter value with the baseline. There are two PLLs on the pcb, LMK3200, and the remainder are in the FPGA. Should there be a disruption in the external clock source to the FEE64 then it is possible the Lock signal from one or more of the PLLs will go false to indicate the PLL is not locked to the input frequency. This transition is counted in the FPGA. If the clock source recovers then this is the only way to understand that a hiatus has occurred.

Noted this morning that the Options file sizes are different and there is no common update date across the 16. I will further improve this function to try and indicate differences ... if required ?

aida07 ADCs will not calibrate. I have attempted to understand why but I have not seen this behaviour before. I suggest, if this is a problem, that a power-cycle be carried out  and then the module is replaced if no improvement is noted.

Merger message logging. I have transferred across my version of the New Merger to the npg folder. /home/npg/Patrick/NewMerger. It runs from the command  /home/npg/Patrick/NewMerger/MergeServer/bin64/run and is currently set for 16 links. The only changes to the NewMerger code have been made in the message.c file. The rest is untouched.

This version will create a log file directory in /MIDAS/log/Merge_Logs. A new directory structure is created here each time the New Merger is started. The directory is named using the date and a number which refers to the number of times the New Merger has been started on that date.  ( /MIDAS/log/Merge_Logs/16_02_22_5 )

Within this directory are text files storing the messages from each of the processes in the New Merger system. The link file messages are stored in a sub-directory as they are currently named by the process number and not the link number. (/MIDAS/log/Merge_Logs/16_02_22_5/links/Link_1529.txt )

The purpose of this change is to be able to correlate error messages reported from the FEE64 consoles with information from the Merger processes.

The system is currently running  with this software for the next fortnight.

  395   Fri Feb 11 12:09:24 2022 NHFri 11 Feb AIDA Work
Use beam break between FRS experiments to finalise getting AIDA testable for PJCS etc.

USB hubs reconnected - all 16 ttyUSBs seem to be visible now

New inverter installed (Ortec 533 Sum Invert, brand new) - Pulser T connected to input B1 and inverted out B sent to -ve FEEs

VME crate for VETAR2 is powered on so White Rabbit should be available

All grounds reconnected on FEEs

All FEEs connected to pulser chain again

-

Moved interlock to new Al shelf on AIDA rack so it's not kicked
Powered on OK, all 4 lights on in box but AIDA Relay had to water light
Investigation shows the cable between relay and interlock was broken, one wire had come loose from connector
Resoldered and now it works as usual

System powered up OK
All 16 FEEs respond to serial
All 16 FEEs have WR
All 16 FEEs show pulser ADC entries

Seems good!
  393   Tue Feb 1 15:00:28 2022 NHAIDA CentOS 7 Startup
Starting up AIDA on CentOS 7 to confirm things work (based on some CARME experience)

NFS settings:
/etc/exports copied over from old disk
line to old /MIDAS_130718 export removed (commented) to see if necessary to copy over
nfsv2 enabled in nfs.conf
NFS service enabled and started (systemctl enable nfs-server ; systemctl start nfs-server)

DHCP setting:
DHCP was disabled, enabled and started (systemctl enable dhcpd ; systemctl start dhcpd)
Raspberry Pis pick up IP and become responsive again

Firwall setting:
Interface p4p1 (AIDA switch #1) is changed to zone 'trusted' so that FEEs can connect to NFS
(panicked before unable to get RPC port)

See FEEs starting to boot!
Getting both / and /MIDAS successfully (note confirming that old /MIDAS_130718 is not needed)

FEEs 4 and 9 take longer to boot others (observed before, seems to be PSU slots 4/5 in first take a while to provide voltage)
FEE9 seems to panic after mounting root, waiting to see if it reboots itself (no serial)
... Nope but power cycle fixes it

All FEEs online and Reset/Setup completed... Temps OK

Note 1: CentOS 7 uses systemd, monitoring log is best done as
`journalctl -f` as root.
npg not in sudoers (should be done)

Note 2: System used as graphical terminal for Edinburgh double-alpha experiment until Feb. 11
AIDA operated in background and using port forwarding to access MIDAS/Web mostly
UI has weird workspace configuration by default needs changing to better cope with AIDA windows

Note 3:
Only 4 FEEs are producing serial data to the Pi_Monitor
Probably 2 of the USB hubs were knocked lose (from the USB extender cables) when the Fatima/HPGe cables were removed last year
S4 access is limited but will try to reconnect ASAP

Note 4:
AIDA setup unfortunately quite limited in functionality due to other running experiments:
WR is unavailable as VETAR VME crate was powered off
Not all FEEs are connected to pulser or grounded. Some cables were removed during testing last year.
Some ground cables removed and pulser inverter removed as needed at CARME and S4 access is now limited for WASA tests
  392   Fri Oct 29 23:00:04 2021 NHGround cable information
For future refinement of the AIDA ground cables using copper bus bar or similar:

Thin cable (from AIDA FEEs to common connection point)
- AWG14 grn/ylw ground cable
- M6 screw crimp?

Thick cable (from common point to mechanical ground)
- AWG4 (25mm2) grn/ylw ground cable
- M12 bolt to mechanical frame 
  391   Fri Oct 29 10:49:01 2021 OHNoise tests - 28th October -29th October
Plan for the day was to connect a mechanical ground to the FEEs connected to DSSD0 1, 2, 3, 4, 9, 10, 11 and 12

This was done by connecting thick copper cable to the mechanical ground wires on the AIDA frame (These ground wires are connected to the platform ground which is earthed properly we are told)
Two thick copper wires were run each one for the top and right side and one for the bottom and left side.
Each cable was connected to the FEEs with a star of smaller cables going to the adapter boards.
Before connection the potential difference between the FEE ground and the mechanical ground was 16mV.
After connection the potential difference was 3-4mV.
It was checked that both the NIM ground and the PSU ground pins were on the same ground as the mechanical ground. It was just the FEE modules on a different ground.
Photos of connection at end of attachments.

It was found later in the afternoon that the 3mV drop comes as a result of the resistance of the bolt used to connect all the thinner wires to the thicker copper braid.

Very good rates observed in all p+n strips but high rates in n+n.
The large oscillations observed in waveforms yesterday are gone though. High frequency component remains
Statistics - attachment 1
Rates spectra - attachment 2
Zoomed in rates spectra - attachment 3
p+n aida11 1.8.W - attachment 4
n+n aida02 1.8.W - attachmeFEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54nt 5

We then reconfigured the PSU so that the FEEs for a given detector were all on the same PSU
Rates are observed to drop across all p+n fees

Statistics - attachment 6
Rates spectra - attachment 7
Rates spectra y max 10 - Attachment 8
p+n aida 12 1.8.W - attachment 9
n+n aida 02 1.8.W - attachment 10

Pulser was then connected to all p+n channels but left off
Rates - attachment 11
Rates spectra - attachment 12
Rates spectra ymax10 - attachment 13

Pulser was then switched on

Rates spectra - attachment 14
Statistics - attachment 15
Pulser peaks p+n - attachment 16

Pulser widths
FEE Width
9   60.25
1   64.18
10  55.76
11  64.81
3   56.21
12  48.54

n=n pulser was then added to both 2 and 4
Pulser peaks n+n - attachment 17
Pulser peaks p+n - attachment 18

FEE Width
9   62.88
1   65.30
10  55.18
11  64.47
3   51.77
12  50.22

2   196.00
4   271.57

29 October

Test adding LK2,3,4 to n+n
n+n all LK1,2,3,4
p+n all lk 2,3,4
Makes p+n side worse - attachment 22

Remove LK3 from n+n
n+n 1,2,4
p+n all LK2,3,4
A slight improvement on previous
attachment 23

Remove LK2 and 4 from n+n
Remove LK3 from all p+n except 1 and 3 which are the two non isolated cables
attachment 24
Attachment 1: 20211028-1300-rates-groundedfees.png
20211028-1300-rates-groundedfees.png
Attachment 2: 20211028-1300-ratespectra.png
20211028-1300-ratespectra.png
Attachment 3: 20211028-1300-ratespectraymax150.png
20211028-1300-ratespectraymax150.png
Attachment 4: 20211028-1300-aida11wave.png
20211028-1300-aida11wave.png
Attachment 5: 20211028-1300-aida02wave.png
20211028-1300-aida02wave.png
Attachment 6: 20211018-1326-rates-psuchange.png
20211018-1326-rates-psuchange.png
Attachment 7: 20211028-1326-ratesspectra-newpsu.png
20211028-1326-ratesspectra-newpsu.png
Attachment 8: 20211028-1326-rates-linymax10-newpsu.png
20211028-1326-rates-linymax10-newpsu.png
Attachment 9: 20211028-1326-wavesaida12-pnpulser-newpsu.png
20211028-1326-wavesaida12-pnpulser-newpsu.png
Attachment 10: 20211028-1326-wavesaida02-newpsupng
Attachment 11: 20211028-1410-ratespulserconnoff.png
20211028-1410-ratespulserconnoff.png
Attachment 12: 20211028-1412-ratespectra-normal.png
20211028-1412-ratespectra-normal.png
Attachment 13: 20211028-1412-ratesspectra-ymax10.png
20211028-1412-ratesspectra-ymax10.png
Attachment 14: 20211028-1413-ratesymax10-pulserONpng
Attachment 15: 20211028-1414-ratesspectra-pulseron.png
20211028-1414-ratesspectra-pulseron.png
Attachment 16: 20211028-1424-pulserpeakspn.png
20211028-1424-pulserpeakspn.png
Attachment 17: 20211028_1449_nn_18L.png
20211028_1449_nn_18L.png
Attachment 18: 20211028_1451_pn_18l.png
20211028_1451_pn_18l.png
Attachment 19: PXL_20211028_121721708.jpg
PXL_20211028_121721708.jpg
Attachment 20: PXL_20211028_121738665.jpg
PXL_20211028_121738665.jpg
Attachment 21: PXL_20211028_121728173.jpg
PXL_20211028_121728173.jpg
Attachment 22: 20211029_1218_stats.png
20211029_1218_stats.png
Attachment 23: 20211029-1400-pnlk1234nnlk124.png
20211029-1400-pnlk1234nnlk124.png
Attachment 24: 20211029-1436-nolk3bar1and3.png
20211029-1436-nolk3bar1and3.png
  390   Wed Oct 27 15:12:50 2021 OH, NHNoise tests - 27th October
 Began by installing the triple.
Only the upstream detector was connected to the FEEs the downstream detector cables were left loose. The two detectors are isotolated though.

Initial cabling configuration
No pulser connections to any of the FEEs connected to the DSSD.
Bias core connected to FEE 13 - FEE 3 and FEE 11 with connections made by T pieces
Bias braid connected to FEE 4 - FEE 2 with connection made by T piece
Adaptor boards grounded to FEE cooling plate via lemo
DSSD cable drain wires also connected to LEMOs

LK1 not on any adaptor boards.
LK2, 3 and 4 on all

System started and rates very high in all of the FEEs connected to DSSD.
N.B FEE 8 connected to pulser but not to cable and adaptor board not grounded to FEE cooling plate
FEE14 connected to pulser and adaptor board grounded to FEE cooling plate
Statistics - Attachment 1
It was noted one of the ribbon cable connectors for FEE13 had come off
Statistics fixed - Attachment 2
Representative waveforms showing the noise - Attachments 3 and 4

Bias turned off and HV filters connected
Issue at first as the solder connection in one filter had broken. Filter was replaced and the DSSD bias
Statistics are unchanged
Waveforms with the HV filters installed - Attachments 5 and 6
Attachment 1: 271021-waveaida10-pndssd-filter.png
271021-waveaida10-pndssd-filter.png
Attachment 2: 271021-dssds-fixed12.png
271021-dssds-fixed12.png
Attachment 3: 271021-waves-dssd-nn.png
271021-waves-dssd-nn.png
Attachment 4: 271021-waveaida10-pndssd.png
271021-waveaida10-pndssd.png
Attachment 5: 271021-waves-aida02-dssdnn-filter.png
271021-waves-aida02-dssdnn-filter.png
Attachment 6: 271021-waveaida10-pndssd-filter.png
271021-waveaida10-pndssd-filter.png
  389   Tue Oct 26 16:08:46 2021 NHFEE and Adaptor board groundings
> During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.
> 
> - The adaptor boards have no connection to the FEE ground over the ERNI connector
>    - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
> - The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
>     - Verified with 0 resistance between plate and HDMI ground shield
> - The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
>     - With electrics disconnected but water still connected they show 200kOhm
> - With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
>     - The cost of this is the rate of FEE8 goes up to 300k
>     - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
> - The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
>    

The adapter boards have ERNI ground pins, and the FEE64 has a grounded plane on its ERNI connector, unfortunately the orientation means the ground pins do not touch the plane on the FEE64 and hence no contact is made
Would require installing board "upside down" but then the ASIC connections would be wrong. A new adapter board layout would be necessary to correct this

Tomorrow will investigate direct connection of all FEE64s to mechanical ground via dedicated copper cable. 
Currently FEE64s get ground via Pulser(->NIM) ((weak ground)), HDMI(->NIM) ((possibly?)) and Power(->PSU) (primary grounding)
HV Bias is floating at NIM side and so does not provide ground path (intentional)
TD may have ideas to isolate pulser from AIDA FEE ground
Unsure if HDMI ground is a problem

The ground quality via the power may not be ideal over long cables of relatively thin wire. (5 * 20 AWG) and then unsure how ground of PSU connects to mechanical ground
Explicit grounding may help keep resistance between FEEs < 1 Ohm.

Will also mount a DSSD and see situation without pulser, with/without mechanical grounding and maybe with various LK jumpers.
  388   Thu Oct 21 16:18:51 2021 OHFEE and Adaptor board groundings
During investigations into the noise today by NH and OH a couple of findings were made on the grounding of the FEEs and Adaptor boards.

- The adaptor boards have no connection to the FEE ground over the ERNI connector
   - Currently we are providing one via a ground cable between the FEE Al cooling and the Lemo ground on the adaptor board
- The Al cooling plate is not electrically isolated from the copper plate on the ASIC and both are connected to the electrical ground on the FEE
    - Verified with 0 resistance between plate and HDMI ground shield
- The FEE is electrically isolated from the frame but when cabling is connected they show 0 resistance
    - With electrics disconnected but water still connected they show 200kOhm
- With the bias core going to FEE14 and the braid going to FEE8 better performance in FEE14 is obtained by disconnecting FEE8 adaptor board from the ground
    - The cost of this is the rate of FEE8 goes up to 300k
    - It is essential to have the adaptor board ground on the FEE ground but unsure why having both adaptor boards connected to their respective grounds makes one worse
- The resistance between all FEEs is not zero. Some show a 5Ohm and some show a 35Ohm resistance between each other
   
  387   Thu Oct 21 09:33:08 2021 OHNoise tests - 21st Oct 2021
Pulser was left on overnight with the bias filters on
First plot is FEE14 not FEE8

Double peaking is observed in both polarities of FEE.
Attachment 1: 211021_0951_fee8_x8L.png
211021_0951_fee8_x8L.png
Attachment 2: 211021_0953_fee14waveform_zoom.png
211021_0953_fee14waveform_zoom.png
Attachment 3: 211021_0956_fee14_waveform.png
211021_0956_fee14_waveform.png
Attachment 4: 211021_0957_fee8_x8L.png
211021_0957_fee8_x8L.png
Attachment 5: 211021_0959_fee8_waveform_zoom.png
211021_0959_fee8_waveform_zoom.png
  386   Thu Oct 21 09:08:41 2021 OHMIDAS start up messages
Attachment 1: 8015Startup.txt
tidy up
tclsh8.5: no process killed

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64
PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin
Computer Name = aidas-gsi; Temp Directory = /tmp/tcl6376
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "kr.j1xicijsz"
httpd started on port 8015

Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8015/startup.tcl
created namespace ::AIDA
/DataBaseAccessServer
/NetVarService
/SigTaskService
loading tcl/AIDARunControl.tcl for namespace ::
RunControlClient provided
DataAcquisitionControlClient
RunControlClient loaded
Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl

Attachment 2: 8115startup
tidy up
tclsh8.5_copy2: no process killed

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64
PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin
Computer Name = aidas-gsi; Temp Directory = /tmp/tcl24260
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "qhsbwphtgqqo"
httpd started on port 8115

Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8115/startup.tcl
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
/TapeServer
loading tcl/NewMergerControl.tcl for namespace ::
/DataAcquisitionControlServer
DefineMessage unknown
Run Control Server Implementation for MERGE
RunControlServer loaded
loading Html/RunControl/implementation.tcl
/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
RunControl loaded
loading Html/NewMerger/RunControl/implementation.tcl for namespace ::
ReadRegister failed: Name=NetVar.MERGE.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
NewMerge Control loaded
Completed custom startup from /MIDAS/TclHttpd/Html/NewMerger/RunControl/stats.defn.tcl

Attachment 3: tapestart.txt
Tidy up
master(4653): Operation not permitted
master: no process killed
stats: no process killed
driver: no process killed
linkTCP: no process killed
Starting Tape Server

[1] 24297
MIDAS Tape Server: Message logger not contacted.
MIDAS Tape Server: MIDAS Tape Server Build Mar 16 2021  12:22:14
MIDAS Tape Server: Unable to change scheduling priority - Permission denied
MIDAS Tape Server: Using default startup
MIDAS Tape Server: Configuration: UDP port = 10205, SHM key=10205.
MIDAS Tape Server: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7f7081936000.
MIDAS Tape Server: Configuration file used - /MIDAS/config/TS_10205/TS_configuration
MIDAS Tape Server: Stats task /MIDAS/TapeServer/Linux64/stats
MIDAS Tape Server: Using device file /dev/file/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device file /dev/file/1 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 10305                 
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 21001 21002                
MIDAS Tape Server: Message reporting level = 0x180fff8
MIDAS Tape Server: Message logging level = 0xfff8
MIDAS Tape Server: Tape Server Options = 0x0
MIDAS Tape Server: File device path base = /TapeData
MIDAS Tape Server: Data buffer size = 65536
MIDAS Tape Server: Tape block size = 65536
MIDAS Tape Server: File mapped object /SHM_110205 of size 4195880 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7f7081535000.
MIDAS Tape Server: File mapped object /SHM_210205 of size 3100 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7f7081a96000.
MIDAS Tape Server: Capabilities restored.
MIDAS Tape Server: Master global area initialised.
MIDAS Tape Server: Stats task has pid 24299
MIDAS Tape Server: Driver process for /dev/file/0 has pid 24300
MIDAS Tape Server: Driver process for /dev/file/1 has pid 24301
MIDAS Tape Server: Driver process for /dev/null/0 has pid 24302
MIDAS Tape Server: Link task 0 has pid 24303
MIDAS Tape Server: Link task 1 has pid 24304
MIDAS Tape Server: Starting the RPC interface
MIDAS Tape Server: Created RPC Program 28000205 Version 4 on UDP port 10205.
MIDAS Tape Server: Entering server loop
MIDAS Tape Server: MIDAS Tape Server now available on UDP port 10205.
MIDAS Tape Statistics: Message logger not contacted.
MIDAS Tape Statistics: MIDAS Tape Statistics Build Mar 16 2021  12:22:20
MIDAS Tape Statistics: Started with args 10205
MIDAS Tape Statistics: Configuration: SHM key=10205
MIDAS Tape Statistics: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Statistics: Shared memory ID is 3
MIDAS Tape Statistics: Shared memory segment located at address 7fb07f300000.
MIDAS Tape Driver (24302): Message logger not contacted.
MIDAS Tape Driver (24302): Message logger not contacted.
MIDAS Tape Driver (24302): Started with args 2 10205
MIDAS Tape Driver (24302): Configuration: driver=2, key=10205.
MIDAS Tape Driver (24302): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (24302): Shared memory ID is 3
MIDAS Tape Driver (24302): Shared memory segment located at address 7f183708f000.
MIDAS Tape Driver (24302): Using device /dev/null/0 of type sink.
MIDAS Tape Driver (24300): Message logger not contacted.
MIDAS Tape Driver (24301): Message logger not contacted.
MIDAS Tape Driver (24300): Message logger not contacted.
MIDAS Tape Driver (24301): Message logger not contacted.
MIDAS Tape Driver (24300): Started with args 0 10205
MIDAS Tape Driver (24301): Started with args 1 10205
MIDAS Tape Driver (24301): Configuration: driver=1, key=10205.
MIDAS Tape Driver (24300): Configuration: driver=0, key=10205.
MIDAS Tape Driver (24301): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (24300): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (24301): Shared memory ID is 3
MIDAS Tape Driver (24300): Shared memory ID is 3
MIDAS Tape Driver (24300): Shared memory segment located at address 7f3b2d8cd000.
MIDAS Tape Driver (24301): Shared memory segment located at address 7ffb5e9ba000.
MIDAS Tape Driver (24300): Using device /dev/file/0 of type file.
MIDAS Tape Driver (24301): Using device /dev/file/1 of type file.
MIDAS Data Link (24304): Message logger not contacted.
MIDAS Data Link (24304): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Data Link (24304): Started with args 10205 21001 21002                
MIDAS Data Link (24304): Configuration: SHM key=10205, TCP port = 21001, Data Stream = 2
MIDAS Data Link (24304): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (24304): Shared memory ID is 3
MIDAS Data Link (24304): Shared memory segment located at address 7f7452b59000.
MIDAS Data Link (24303): Message logger not contacted.
MIDAS Data Link (24304): Starting the network interface
MIDAS Data Link (24303): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Data Link (24303): Started with args 10205 10305                 
MIDAS Data Link (24303): Configuration: SHM key=10205, TCP port = 10305, Data Stream = 1
MIDAS Data Link (24303): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (24303): Shared memory ID is 3
MIDAS Data Link (24303): Shared memory segment located at address 7f754106b000.
MIDAS Data Link (24303): Starting the network interface
MIDAS Data Link (24303): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (24303): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (24303): MIDAS Data Link thread 0 using TCP port 10305.
MIDAS Data Link (24303): Entering server loop
MIDAS Data Link (24303): thread 0 listening on port 10305
MIDAS Data Link (24304): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (24304): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (24304): MIDAS Data Link thread 0 using TCP port 21001.
MIDAS Data Link (24304): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (24304): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (24304): MIDAS Data Link thread 1 using TCP port 21002.
MIDAS Data Link (24304): Entering server loop
MIDAS Data Link (24304): thread 0 listening on port 21001
MIDAS Data Link (24304): thread 1 listening on port 21002

Attachment 4: mergerstart.txt
Tidy up
MERGE: no process killed
merge64.AD: no process killed
master64: no process killed
link64: no process killed
merge64: no process killed
statrate64: no process killed
Starting New Merger

[1] 24325
MERGE Master (24325): Message logger not contacted.
MERGE Master (24325): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
Creating NetVars
Creating NetVars complete
MERGE Master (24325): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
MERGE Master (24325): Configuration: TCP port base=11001 ; SHM key=11000; ids=16; links=16; Hardware=0
MERGE Master (24325): File mapped object /SHM_11000 of size 536628 created
MERGE Master (24325): Memory mapping 536628 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5efe67a000.
MERGE Master (24325): File mapped object /SHM_11001 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ef8678000.
MERGE Master (24325): File mapped object /SHM_11002 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ef2676000.
MERGE Master (24325): File mapped object /SHM_11003 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eec674000.
MERGE Master (24325): File mapped object /SHM_11004 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ee6672000.
MERGE Master (24325): File mapped object /SHM_11005 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ee0670000.
MERGE Master (24325): File mapped object /SHM_11006 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eda66e000.
MERGE Master (24325): File mapped object /SHM_11007 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ed466c000.
MERGE Master (24325): File mapped object /SHM_11008 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ece66a000.
MERGE Master (24325): File mapped object /SHM_11009 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ec8668000.
MERGE Master (24325): File mapped object /SHM_11010 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ec2666000.
MERGE Master (24325): File mapped object /SHM_11011 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ebc664000.
MERGE Master (24325): File mapped object /SHM_11012 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eb6662000.
MERGE Master (24325): File mapped object /SHM_11013 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eb0660000.
MERGE Master (24325): File mapped object /SHM_11014 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5eaa65e000.
MERGE Master (24325): File mapped object /SHM_11015 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5ea465c000.
MERGE Master (24325): File mapped object /SHM_11016 of size 100669480 created
MERGE Master (24325): Memory mapping 100669480 bytes
MERGE Master (24325): Shared memory segment located at address 0x7f5e9e65a000.
MERGE Master (24325): Shared memory segment located at address 0x7f5e9e65a000.
MERGE Master (24325): Starting stats/rates task
MERGE Master (24325): Stats/Rates task has pid 24327
MERGE Master (24325): Starting 16 link tasks
MERGE Master (24325): Link task has pid 24328
MERGE Master (24325): Link task has pid 24329
MERGE Master (24325): Link task has pid 24330
MERGE Master (24325): Link task has pid 24331
MERGE Master (24325): Link task has pid 24332
MERGE Master (24325): Link task has pid 24333
MERGE Master (24325): Link task has pid 24334
MERGE Master (24325): Link task has pid 24335
MERGE Master (24325): Link task has pid 24336
MERGE Master (24325): Link task has pid 24337
MERGE Statistics/Rates (24327): Message logger not contacted.
MERGE Statistics/Rates (24327): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
Creating NetVars
Creating NetVars complete
MERGE Master (24325): Link task has pid 24338
MERGE Master (24325): Link task has pid 24339
MERGE Master (24325): Link task has pid 24340
MERGE Master (24325): Link task has pid 24341
MERGE Master (24325): Link task has pid 24342
MERGE Master (24325): Link task has pid 24343
MERGE Data Link (24329): Message logger not contacted.
MERGE Data Link (24330): Message logger not contacted.
MERGE Data Link (24330): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24329): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars
Creating NetVars complete
MERGE Data Link (24330): Configuration: index = 2, SHM key=11000, TCP port base = 11003, module = 1, hardware = 0
MERGE Data Link (24330): File mapped object /SHM_11000 accessed
MERGE Data Link (24330): Memory mapping 536628 bytes
MERGE Data Link (24330): Shared memory segment located at address 0x7f6556fb6000.
MERGE Data Link (24330): Allocated 71720 bytes for local input buffer
MERGE Data Link (24330): File mapped object /SHM_11003 accessed
MERGE Data Link (24330): Memory mapping 100669480 bytes
MERGE Data Link (24330): Shared memory segment located at address 0x7f6550fb4000.
MERGE Data Link (24328): Message logger not contacted.
MERGE Data Link (24328): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24328): Configuration: index = 0, SHM key=11000, TCP port base = 11001, module = -1, hardware = 0
MERGE Data Link (24328): File mapped object /SHM_11000 accessed
MERGE Data Link (24328): Memory mapping 536628 bytes
MERGE Data Link (24328): Shared memory segment located at address 0x7efe94f4e000.
MERGE Data Link (24328): Allocated 71720 bytes for local input buffer
MERGE Data Link (24328): File mapped object /SHM_11001 accessed
MERGE Data Link (24328): Memory mapping 100669480 bytes
MERGE Data Link (24328): Shared memory segment located at address 0x7efe8ef4c000.
MERGE Data Link (24336): Message logger not contacted.
MERGE Data Link (24336): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24336): Configuration: index = 8, SHM key=11000, TCP port base = 11009, module = 7, hardware = 0
MERGE Data Link (24336): File mapped object /SHM_11000 accessed
MERGE Data Link (24336): Memory mapping 536628 bytes
MERGE Data Link (24336): Shared memory segment located at address 0x7f020eeee000.
MERGE Data Link (24336): Allocated 71720 bytes for local input buffer
MERGE Data Link (24336): File mapped object /SHM_11009 accessed
MERGE Data Link (24336): Memory mapping 100669480 bytes
MERGE Data Link (24336): Shared memory segment located at address 0x7f0208eec000.
MERGE Data Link (24331): Message logger not contacted.
MERGE Data Link (24331): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24331): Configuration: index = 3, SHM key=11000, TCP port base = 11004, module = 2, hardware = 0
MERGE Data Link (24331): File mapped object /SHM_11000 accessed
MERGE Data Link (24331): Memory mapping 536628 bytes
MERGE Data Link (24331): Shared memory segment located at address 0x7f44daa69000.
MERGE Data Link (24331): Allocated 71720 bytes for local input buffer
MERGE Data Link (24331): File mapped object /SHM_11004 accessed
MERGE Data Link (24331): Memory mapping 100669480 bytes
MERGE Data Link (24331): Shared memory segment located at address 0x7f44d4a67000.
MERGE Data Link (24337): Message logger not contacted.
MERGE Data Link (24337): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (24334): Message logger not contacted.
MERGE Data Link (24334): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24343): Message logger not contacted.
Creating NetVars
MERGE Data Link (24343): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (24333): Message logger not contacted.
MERGE Data Link (24333): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (24335): Message logger not contacted.
MERGE Data Link (24342): Message logger not contacted.
MERGE Data Link (24342): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24335): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars
MERGE Data Link (24341): Message logger not contacted.
Creating NetVars complete
MERGE Data Link (24337): Configuration: index = 9, SHM key=11000, TCP port base = 11010, module = 8, hardware = 0
MERGE Data Link (24337): File mapped object /SHM_11000 accessed
MERGE Data Link (24341): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24337): Memory mapping 536628 bytes
MERGE Data Link (24337): Shared memory segment located at address 0x7f12856cb000.
Creating NetVars
Creating NetVars complete
MERGE Data Link (24334): Configuration: index = 6, SHM key=11000, TCP port base = 11007, module = 5, hardware = 0
MERGE Data Link (24334): File mapped object /SHM_11000 accessed
MERGE Data Link (24334): Memory mapping 536628 bytes
MERGE Data Link (24332): Message logger not contacted.
MERGE Data Link (24334): Shared memory segment located at address 0x7fb1348e4000.
MERGE Data Link (24337): Allocated 71720 bytes for local input buffer
MERGE Data Link (24337): File mapped object /SHM_11010 accessed
MERGE Data Link (24332): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24337): Memory mapping 100669480 bytes
MERGE Data Link (24340): Message logger not contacted.
MERGE Data Link (24337): Shared memory segment located at address 0x7f127f6c9000.
Creating NetVars
MERGE Data Link (24340): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (24334): Allocated 71720 bytes for local input buffer
Creating NetVars
MERGE Data Link (24334): File mapped object /SHM_11007 accessed
MERGE Data Link (24334): Memory mapping 100669480 bytes
MERGE Data Link (24334): Shared memory segment located at address 0x7fb12e8e2000.
MERGE Data Link (24339): Message logger not contacted.
MERGE Data Link (24339): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24333): Configuration: index = 5, SHM key=11000, TCP port base = 11006, module = 4, hardware = 0
Creating NetVars complete
MERGE Data Link (24333): File mapped object /SHM_11000 accessed
MERGE Data Link (24333): Memory mapping 536628 bytes
MERGE Data Link (24335): Configuration: index = 7, SHM key=11000, TCP port base = 11008, module = 6, hardware = 0
MERGE Data Link (24333): Shared memory segment located at address 0x7f3a30878000.
MERGE Data Link (24335): File mapped object /SHM_11000 accessed
MERGE Data Link (24335): Memory mapping 536628 bytes
MERGE Data Link (24335): Shared memory segment located at address 0x7fd567c53000.
MERGE Data Link (24333): Allocated 71720 bytes for local input buffer
MERGE Data Link (24335): Allocated 71720 bytes for local input buffer
MERGE Data Link (24335): File mapped object /SHM_11008 accessed
MERGE Data Link (24335): Memory mapping 100669480 bytes
MERGE Data Link (24335): Shared memory segment located at address 0x7fd561c51000.
MERGE Data Link (24333): File mapped object /SHM_11006 accessed
MERGE Data Link (24333): Memory mapping 100669480 bytes
MERGE Data Link (24333): Shared memory segment located at address 0x7f3a2a876000.
Creating NetVars complete
MERGE Data Link (24341): Configuration: index = 13, SHM key=11000, TCP port base = 11014, module = 12, hardware = 0
MERGE Data Link (24341): File mapped object /SHM_11000 accessed
MERGE Data Link (24341): Memory mapping 536628 bytes
MERGE Data Link (24341): Shared memory segment located at address 0x7f84b1fef000.
MERGE Data Link (24341): Allocated 71720 bytes for local input buffer
MERGE Data Link (24341): File mapped object /SHM_11014 accessed
MERGE Data Link (24341): Memory mapping 100669480 bytes
MERGE Data Link (24341): Shared memory segment located at address 0x7f84abfed000.
Creating NetVars complete
Creating NetVars complete
MERGE Data Link (24332): Configuration: index = 4, SHM key=11000, TCP port base = 11005, module = 3, hardware = 0
MERGE Data Link (24340): Configuration: index = 12, SHM key=11000, TCP port base = 11013, module = 11, hardware = 0
MERGE Data Link (24332): File mapped object /SHM_11000 accessed
MERGE Data Link (24332): Memory mapping 536628 bytes
MERGE Data Link (24332): Shared memory segment located at address 0x7fc8787db000.
MERGE Data Link (24340): File mapped object /SHM_11000 accessed
MERGE Data Link (24340): Memory mapping 536628 bytes
MERGE Data Link (24340): Shared memory segment located at address 0x7f9523835000.
Creating NetVars complete
MERGE Data Link (24339): Configuration: index = 11, SHM key=11000, TCP port base = 11012, module = 10, hardware = 0
MERGE Data Link (24332): Allocated 71720 bytes for local input buffer
MERGE Data Link (24340): Allocated 71720 bytes for local input buffer
MERGE Data Link (24339): File mapped object /SHM_11000 accessed
MERGE Data Link (24332): File mapped object /SHM_11005 accessed
MERGE Data Link (24339): Memory mapping 536628 bytes
MERGE Data Link (24332): Memory mapping 100669480 bytes
MERGE Data Link (24339): Shared memory segment located at address 0x7ff8a2e8c000.
MERGE Data Link (24332): Shared memory segment located at address 0x7fc8727d9000.
MERGE Data Link (24340): File mapped object /SHM_11013 accessed
MERGE Data Link (24340): Memory mapping 100669480 bytes
MERGE Data Link (24340): Shared memory segment located at address 0x7f951d833000.
MERGE Data Link (24339): Allocated 71720 bytes for local input buffer
MERGE Data Link (24339): File mapped object /SHM_11012 accessed
MERGE Data Link (24339): Memory mapping 100669480 bytes
MERGE Data Link (24339): Shared memory segment located at address 0x7ff89ce8a000.
MERGE Data Link (24338): Message logger not contacted.
MERGE Data Link (24338): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (24338): Configuration: index = 10, SHM key=11000, TCP port base = 11011, module = 9, hardware = 0
MERGE Data Link (24338): File mapped object /SHM_11000 accessed
MERGE Data Link (24338): Memory mapping 536628 bytes
MERGE Data Link (24338): Shared memory segment located at address 0x7f62f5191000.
MERGE Data Link (24338): Allocated 71720 bytes for local input buffer
MERGE Data Link (24338): File mapped object /SHM_11011 accessed
MERGE Data Link (24338): Memory mapping 100669480 bytes
MERGE Data Link (24338): Shared memory segment located at address 0x7f62ef18f000.
MERGE Statistics/Rates (24327): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
MERGE Statistics thread starting
MERGE Data Link (24330): Shared memory segment located at address 0x7f6550fb4000.
MERGE Data Link (24330): Shared memory segment located at address 0x7f6550fb4000.
MERGE Data Link (24335): Shared memory segment located at address 0x7fd561c51000.
MERGE Data Link (24335): Shared memory segment located at address 0x7fd561c51000.
MERGE Data Link (24338): Shared memory segment located at address 0x7f62ef18f000.
MERGE Data Link (24338): Shared memory segment located at address 0x7f62ef18f000.
MERGE Data Link (24336): Shared memory segment located at address 0x7f0208eec000.
MERGE Data Link (24336): Shared memory segment located at address 0x7f0208eec000.
MERGE Data Link (24328): Shared memory segment located at address 0x7efe8ef4c000.
MERGE Data Link (24328): Shared memory segment located at address 0x7efe8ef4c000.
MERGE Data Link (24331): Shared memory segment located at address 0x7f44d4a67000.
MERGE Data Link (24331): Shared memory segment located at address 0x7f44d4a67000.
MERGE Data Link (24341): Shared memory segment located at address 0x7f84abfed000.
MERGE Data Link (24341): Shared memory segment located at address 0x7f84abfed000.
MERGE Data Link (24334): Shared memory segment located at address 0x7fb12e8e2000.
MERGE Data Link (24334): Shared memory segment located at address 0x7fb12e8e2000.
MERGE Data Link (24339): Shared memory segment located at address 0x7ff89ce8a000.
MERGE Data Link (24339): Shared memory segment located at address 0x7ff89ce8a000.
MERGE Data Link (24332): Shared memory segment located at address 0x7fc8727d9000.
MERGE Data Link (24332): Shared memory segment located at address 0x7fc8727d9000.
MERGE Data Link (24337): Shared memory segment located at address 0x7f127f6c9000.
MERGE Data Link (24337): Shared memory segment located at address 0x7f127f6c9000.
MERGE Data Link (24340): Shared memory segment located at address 0x7f951d833000.
MERGE Data Link (24340): Shared memory segment located at address 0x7f951d833000.
MERGE Data Link (24333): Shared memory segment located at address 0x7f3a2a876000.
MERGE Data Link (24333): Shared memory segment located at address 0x7f3a2a876000.

Attachment 5: datarelay.txt
Tidy up
DataRelayMBS: no process killed
Starting Data Relay to MBS

Setting Overlap Mode 0
Setting Blocking Mode 0
Setting nice 1
Setting Endian 1
Setting Transfer Mode 4
Setting Transfer Block Size 65536
TCP transfer library version 4.0
0: TCP socket send buffer was 16384 - now 249856
0: TCP socket receive buffer was 87380 - now 249856
0: TCP socket created OK - now connecting to x86l-94 port 6500
Transfer Error - : Connection refused
Error 111
0: connect() failed:
Installing signal handlers for MBS connection
Unable to set signal handler for 9
Unable to set signal handler for 19
Unable to set signal handler for 32
Unable to set signal handler for 33
sigtrap 0
sigtrap 0
TCP transfer library version 4.0
0: TCP socket send buffer was 16384 - now 249856
0: TCP socket receive buffer was 87380 - now 249856
0: TCP socket created OK - now connecting to x86l-94 port 6500
Transfer Error - : Connection refused
Error 111
0: connect() failed:
Sent Start to MBS with code=-1
sigtrap 0
dataSpy Shared buffer area 0 (/SHM_110205) located at 0x6a586000
dataSpy Current age 0 index 0
Spy open = 0
  385   Wed Oct 20 13:26:08 2021 NH, OHAIDA Noise + Filtering
Turn on AIDA (SL6) to investigate noise situation and impact of new RC filters on bias input

Current setup:
- No Detectors
- Pulser in fee14, fee8 *only*, no bias connected. FEEs isolated from each other

Pulser widths::
p+n grounds connected via bias link (no connection to HV bias itself)

ASICFEE14 WidthFEE8 Width
1.8.L14.4121.86
2.8.L13.0221.87
3.8.L12.5721.52
4.8.L13.7523.59


Note: FEE8 is connected via SA1 inverter -> cause of worse resolution

Plug bias into outer FEEs (14 + 8 + chaint o 6)

ASICFEE14 WidthFEE8 Width
1.8.L14.5924.43
2.8.L12.7322.25
3.8.L12.7623.06
4.8.L14.3623.31


Long run over lunch:

ASICFEE14 WidthFEE8 Width
1.8.L14.4725.52
2.8.L12.8324.26
3.8.L12.7524.15
4.8.L14.5826.93

Figures 1 & 2

Add HV filters

ASICFEE14 WidthFEE8 Width
1.8.L14.3423.53
2.8.L12.6723.82
3.8.L12.7823.59
4.8.L13.8825.76

Figures 3 & 4

Small improvement maybe?

Figs 5&6 - Oscilloscope before & after HV filter added

--

Plans for tomorrow (21/10/21)

- Reintroduce pulser cables through all FEEs to see impact on noise (maybe improved now)
- Measure pulser noise in battery DSO (+ other sourceS?)
- Add a DSSD and see impact on noise with HV filter
- Resistance from FEE64 adapter to mechanical ground
- Check pulser on the battery powered scope (Is the signal common to the power supply)
Attachment 1: 211020_1316_fee14_bias_con_long_stats.png
211020_1316_fee14_bias_con_long_stats.png
Attachment 2: 211020_1314_fee8_bias_con_long_stat.png
211020_1314_fee8_bias_con_long_stat.png
Attachment 3: 211020_1421_fee14_bias_con_filtered_long.png
211020_1421_fee14_bias_con_filtered_long.png
Attachment 4: 211020_1415_fee8_bias_con_filtered_long.png
211020_1415_fee8_bias_con_filtered_long.png
Attachment 5: scope_before.png
scope_before.png
Attachment 6: scope_after.png
scope_after.png
  384   Tue Oct 19 16:17:49 2021 OHAIDA-gsi server to centos 7
Following advice of CU have updated the server to centos 7.

To do this a new 500GB SSD was purchased and installed in the enclosure and a clean install of centos7 was installed to this.
An npg account was setup with the same password as previously. Likewise the su password is the same

The existing 500GB HDD can then be mounted using the disk tool in Applications/Utilities/Disks (Requires administrator rights)

The MIDAS_Releases directory was tarballed and coppied across to its respective place on the new install.
A symlink from /MIDAS to this location was created

dhcpd.conf was copied across. dhcp needed to be installed on the machine as well.
/etc/hosts was also required to get the hostnames for the rpis working

oh, GREAT, LayOut, Patrick and analyser directories were also tarballed and moved across

.desktop files have been created for all of the shell scripts used to start the servers.
A gnome extension Frippery panel favourites has also been installed which allows the shortcuts to be stored in the top panel similar to sl6

Early tests appear to be working.

Will test with the FEEs later this week.

Currently the machine is booted into centos but will be booting into sl6 tomorrow to run some tests.
  383   Mon Oct 18 14:20:32 2021 OHTop bar launchers
launcher1
TclHttpd @8015 for AIDA
/MIDAS/Linux/startup/HTTPD

launcher2
TclHttpd @8115 for AIDA
/MIDAS/Linux/startup/HTTPD@8115

launcher3
TapeServer
/MIDAS/Linux/startup/TapeServer

launcher4
New Merger for AIDA
/MIDAS/Linux/startup/NewMerger

Launcher5
MBS Relay
/MIDAS/Linux/startup/datarelaymbs
  382   Mon Oct 18 13:03:23 2021 OHDHCPd.conf backup
Attachment 1: dhcpd.conf
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
# Date of last update  Jan 12 2015 
#
authoritative;

ddns-update-style none; ddns-updates off;

# 2 days
#default-lease-time 172800;
# 4 days
default-lease-time 345600;
# 8 days
max-lease-time 691200;

option domain-search code 119 = string;

option domain-name "dl.ac.uk";
option domain-name-servers 193.62.115.16, 148.79.80.78;
option netbios-name-servers 148.79.160.89;
option netbios-node-type 8;
option nis-domain "nuclear.physics";
option nis-servers 193.62.115.77;


subnet 192.168.11.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.11.255;
   pool {
 range 192.168.11.118 192.168.11.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.11.99";
   next-server 192.168.11.99;

   host nnrpi1 {
     hardware ethernet b8:27:eb:bb:46:7b;
     fixed-address 192.168.11.251;
   }

   host nnrpi2 {
     hardware ethernet b8:27:eb:40:53:e8;
     fixed-address 192.168.11.117;
   }

   host aida01 {
     hardware ethernet d8:80:39:41:ba:8a;
     fixed-address 192.168.11.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01";
   }

   host aida02 {
     hardware ethernet d8:80:39:41:ba:22;
     fixed-address 192.168.11.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02";
   }

   host aida03 {
     hardware ethernet d8:80:39:41:d8:21;
     fixed-address 192.168.11.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03";
   }

   host aida04 {
     hardware ethernet d8:80:39:41:f6:b7;
     fixed-address 192.168.11.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04";
   }

   host aida05 {
     hardware ethernet d8:80:39:41:d7:cc;
     fixed-address 192.168.11.5;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   }

   host aida06 {
     hardware ethernet d8:80:39:41:ee:72;
     fixed-address 192.168.11.6;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   }

   host aida07 {
     hardware ethernet d8:80:39:41:b4:0c;
     fixed-address 192.168.11.7;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   }

   host aida08 {
     hardware ethernet d8:80:39:41:ba:2b;
     fixed-address 192.168.11.8;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   }

   host aida09 {
     hardware ethernet d8:80:39:41:f6:ee;
     fixed-address 192.168.11.9;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09";
   }

   host aida10 {
     hardware ethernet d8:80:39:41:ba:89;
     fixed-address 192.168.11.10;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10";
   }

   host aida11 {
     hardware ethernet d8:80:39:41:f6:5a;
     fixed-address 192.168.11.11;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11";
   }

   host aida12 {
     hardware ethernet d8:80:39:41:cf:ac;
     fixed-address 192.168.11.12;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12";
   }

   host aida13 {
     hardware ethernet d8:80:39:42:d:15;
     fixed-address 192.168.11.13;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida13";
   }

   host aida14 {
     hardware ethernet d8:80:39:42:d:b;
     fixed-address 192.168.11.14;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14";
   }

   host aida15 {
     hardware ethernet d8:80:39:41:ee:10;
     fixed-address 192.168.11.15;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15";
   }

   host aida16 {
     hardware ethernet d8:80:39:41:f6:ed;
     fixed-address 192.168.11.16;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida16";
   }
}


subnet 192.168.12.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.12.255;
   pool {
 range 192.168.12.100 192.168.12.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.12.99";
   next-server 192.168.12.99;

   host aida21 {
     hardware ethernet d8:80:39:41:d8:2a;
     fixed-address 192.168.12.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida21";
   }

   host aida22 {
     hardware ethernet 00:04:a3:2a:d0:26;
     fixed-address 192.168.12.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida22";
   }

   host aida23 {
     hardware ethernet 00:04:a3:2b:09:ce;
     fixed-address 192.168.12.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida23";
   }

   host aida24 {
     hardware ethernet 00:04:a3:2b:09:e8;
     fixed-address 192.168.12.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida24";
   }

}



  381   Mon Oct 18 13:01:04 2021 OHProxy config
Proxy for firefox and anydesk
Attachment 1: proxy.png
proxy.png
Attachment 2: proxy2.png
proxy2.png
  380   Fri Sep 3 13:39:42 2021 OHAdditional noise tests performed at GSI
Contents of an email from NH to OH and TD about tests performed on an additional bias supply:

This email was sent to you by someone outside the University.
You should only click on links or attachments if you are certain that the email is genuine and the content is safe.

so I looked at the other supply in the lab (in a NIM crate there, cable direct to DSO)

Still see the same noise wave - also it looked a lot worse at 50V but that may be a worse NIM crate or something else - the current also wasn't 0 (calibration?)

More unusually the signals remain even if the NIM crate is off (DSO_Lab_NoP) and when the SHV is removed from the HV completely (DSO_Lab_NoCable)
Attachment 1: DSO_Lab_NoP.jpg
DSO_Lab_NoP.jpg
Attachment 2: DSO_Lab_1.jpg
DSO_Lab_1.jpg
Attachment 3: DSO_Lab_50V.jpg
DSO_Lab_50V.jpg
Attachment 4: DSO_Lab_NoCable.jpg
DSO_Lab_NoCable.jpg
  379   Sun Aug 22 18:40:08 2021 TDAIDA Noise
> > 
> > 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
> >       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
> >       Switch Cooknell SA1 Sum Amp output from negative to positive
> > 
> >       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> > 
> >       This probably indicates
> >       - noisy Cooknell SA1
> >       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> > 
> > > Investigate noise situation of AIDA (no DSSSDs connected)
> > > 
> > > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > > - back-side aidas (02,04,06,08) worse resolution than front-side
> > > - high-frequency noise still noticeable in waveforms
> > > 
> > > Figs 7-8: Remove pulser in (from NIM rack) only
> > > - no noticable change
> > > 
> > > Figs 9-10: Remove all pulser cables from adapter boards
> > > - looks improved
> > > 
> > > Figs 11-12: HV Braid->GND Jumpers removed
> > > 
> > > Fig 13-14: All HV cables removed
> > > - aida06, aida14 isn't updating
> > > - very noticable improvement
> > > 
> > > Fig 15-16: Intermediate HV cables re-added
> > > - signal worse in FEE64s with HV cables
> > > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > > 
> > > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > > 
> > > Pulser widths now:
> > > - aida14 (-ve amplitude) = 13.5 channels
> > > - aida08 (+ve amplitude) = 40 channels (?)
> 
> Removed the T-piece between pulser and summing amplifier
> pulser FWHM still ~30 channels - not the T-piece, probably summing amp
> 
> Checked the noise of HV and LV on battery DSO
> - LV - no noticeable noise on any channel (< 5 mV rms)
> - HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
>        independent of bias voltage 
>        See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
>                       AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2
> 
> Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
> 100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
> Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 

1nF/3k => 333kHz low pass rolloff

10nF/1M => 100Hz low pass rolloff

>  Check on one DSSSD, if see improvement can migrate to adapter boards in new revision
> 
> Things to do:
> TD:
>  Inline filters
> NH:
>  Check same behaviour other N1419ET HV supplies
>  Check line voltages from CAEN NIM bin
  378   Fri Aug 20 14:33:24 2021 TDAIDA Noise
> 
> 11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
>       Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
>       Switch Cooknell SA1 Sum Amp output from negative to positive
> 
>       Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14
> 
>       This probably indicates
>       - noisy Cooknell SA1
>       - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1
> 
> > Investigate noise situation of AIDA (no DSSSDs connected)
> > 
> > Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> > - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> > - back-side aidas (02,04,06,08) worse resolution than front-side
> > - high-frequency noise still noticeable in waveforms
> > 
> > Figs 7-8: Remove pulser in (from NIM rack) only
> > - no noticable change
> > 
> > Figs 9-10: Remove all pulser cables from adapter boards
> > - looks improved
> > 
> > Figs 11-12: HV Braid->GND Jumpers removed
> > 
> > Fig 13-14: All HV cables removed
> > - aida06, aida14 isn't updating
> > - very noticable improvement
> > 
> > Fig 15-16: Intermediate HV cables re-added
> > - signal worse in FEE64s with HV cables
> > - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> > 
> > Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> > 
> > Pulser widths now:
> > - aida14 (-ve amplitude) = 13.5 channels
> > - aida08 (+ve amplitude) = 40 channels (?)

Removed the T-piece between pulser and summing amplifier
pulser FWHM still ~30 channels - not the T-piece, probably summing amp

Checked the noise of HV and LV on battery DSO
- LV - no noticeable noise on any channel (< 5 mV rms)
- HV - noticable periodic signal (~100 kHz, 100 mV Vpp) on both core and braid channels, common to both
       independent of bias voltage 
       See figs 1-6 : 0V, 10V, 20V, 50V, 100V, 0V (high photo resolution) of bias voltage
                      AC coupled 1MOhm Ch1 = HV Core, Ch2= HV Braid, Math=Ch1-Ch2

Filtering on AIDA front end ~ 10 nF (gnd), 3 kOhm  (series) - low pass ~ 33kHz rolloff
100 kHz would be attenuated factor of 5 maybe - 20 mV signal - still quite high for ASICs
Try introducing additional inline filtering of maybe 3 kHz (1 nF gnd, 3 kOhm series) 
 Check on one DSSSD, if see improvement can migrate to adapter boards in new revision

Things to do:
TD:
 Inline filters
NH:
 Check same behaviour other N1419ET HV supplies
 Check line voltages from CAEN NIM bin
Attachment 1: aida_0v.jpg
aida_0v.jpg
Attachment 2: aida_10v.jpg
aida_10v.jpg
Attachment 3: aida_20v.jpg
aida_20v.jpg
Attachment 4: aida_50v.jpg
aida_50v.jpg
Attachment 5: aida_100v.jpg
aida_100v.jpg
Attachment 6: aida_0v.jpg
aida_0v.jpg
  377   Thu Aug 19 10:47:13 2021 TDMIDAS spectra
Attachment 1: AIDA_spectra_190821.tar.gz
  376   Thu Aug 19 10:37:50 2021 TDAIDA Noise
11:30 BNC PB-5 pulser polarity chnaged from positive to negative (using LOCAL control)
      Swap BNC cables (top + bottom & left + right) from BNC PB-5 and Cooknell SA1 Sum Amp outputs
      Switch Cooknell SA1 Sum Amp output from negative to positive

      Observe pulser peak of c. 15 ch FWHM for aida08 and c. 35 ch FWHM for aida14

      This probably indicates
      - noisy Cooknell SA1
      - problem with T and/or Lemo-00 cable between BNC PB-5 and Cooknell SA1

> Investigate noise situation of AIDA (no DSSSDs connected)
> 
> Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
> - aida16 very noisy ASICS 1&2, normal ASICS 3&4
> - back-side aidas (02,04,06,08) worse resolution than front-side
> - high-frequency noise still noticeable in waveforms
> 
> Figs 7-8: Remove pulser in (from NIM rack) only
> - no noticable change
> 
> Figs 9-10: Remove all pulser cables from adapter boards
> - looks improved
> 
> Figs 11-12: HV Braid->GND Jumpers removed
> 
> Fig 13-14: All HV cables removed
> - aida06, aida14 isn't updating
> - very noticable improvement
> 
> Fig 15-16: Intermediate HV cables re-added
> - signal worse in FEE64s with HV cables
> - aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good
> 
> Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.
> 
> Pulser widths now:
> - aida14 (-ve amplitude) = 13.5 channels
> - aida08 (+ve amplitude) = 40 channels (?)
  375   Tue Aug 17 12:39:25 2021 NH, TDAIDA Noise
Investigate noise situation of AIDA (no DSSSDs connected)

Figs 1-6: Waves, Pulser Peaks + Widths *before* any changes
- aida16 very noisy ASICS 1&2, normal ASICS 3&4
- back-side aidas (02,04,06,08) worse resolution than front-side
- high-frequency noise still noticeable in waveforms

Figs 7-8: Remove pulser in (from NIM rack) only
- no noticable change

Figs 9-10: Remove all pulser cables from adapter boards
- looks improved

Figs 11-12: HV Braid->GND Jumpers removed

Fig 13-14: All HV cables removed
- aida06, aida14 isn't updating
- very noticable improvement

Fig 15-16: Intermediate HV cables re-added
- signal worse in FEE64s with HV cables
- aida09, aida01, aida10, aida13, aida05, aida14 don't have HV interconnect cables, look good

Conclusion: We see noticeable improvements removing the interconnect cables between FEE64s.

Pulser widths now:
- aida14 (-ve amplitude) = 13.5 channels
- aida08 (+ve amplitude) = 40 channels (?)
Attachment 1: Screenshot-Waves1.png
Screenshot-Waves1.png
Attachment 2: Screenshot-Waves2.png
Screenshot-Waves2.png
Attachment 3: Screenshot-Pulsers.png
Screenshot-Pulsers.png
Attachment 4: Screenshot-Width11.png
Screenshot-Width11.png
Attachment 5: Screenshot-PulserWidthBack.png
Screenshot-PulserWidthBack.png
Attachment 6: Screenshot-Width02.png
Screenshot-Width02.png
Attachment 7: Screenshot-PulserInRemov-Front.png
Screenshot-PulserInRemov-Front.png
Attachment 8: Screenshot-PulserInRemov-Back.png
Screenshot-PulserInRemov-Back.png
Attachment 9: PulserAllOut-Front.png
PulserAllOut-Front.png
Attachment 10: PulserAllremov-Back.png
PulserAllremov-Back.png
Attachment 11: HVJumpers-Front.png
HVJumpers-Front.png
Attachment 12: HVJumpers-Back.png
HVJumpers-Back.png
Attachment 13: AllHVRemoved-Front.png
AllHVRemoved-Front.png
Attachment 14: HVout-Back.png
HVout-Back.png
Attachment 15: HVintermediate-Front.png
HVintermediate-Front.png
Attachment 16: HVintermediate-Back.png
HVintermediate-Back.png
  374   Mon Jul 5 14:36:15 2021 NHMonday 5 July
AIDA Unbiased and powered down
All FEEs were running before stoppage. None had rebooted.
Aida05 did not respond until merger was restarted, all other DAQs responded fine.

Snout removed from S4 and downstream bPlast removed from snout to go into Scanner setup
AIDA will remain in Snout in lab for time being.

FEEs will be reconnected soon to run in pulser-only mode if desired - but kept powered off for now
  373   Tue Jun 22 13:01:19 2021 NHAIDA Data on lustre
The following runs are now copied to lustre
If necessary they can be removed from AIDA HDDs for space.
There are also spare HDDs that can (and should) be swapped for the current ones

/lustre/gamma/DESPEC_S452_FILES/AIDA
S452
R1
R2
R3
R4
R5
R6
R7
R9
R11
R13
R14
R15
R16
R17
R20
R21
R22
R23
R25
R27
R28
R29
R30
R33
R34
R35
R36
R38
R39
R40
R42
R43
R46
R50
R56
R57
R58

/lustre/gamma/DESPEC_S460_FILES/AIDA
S460
R1
R2
R4
R5
R7
R9
R11
R13
R14
R16
R20
R21
R23
R24
R25
R26
R27
R30
R31
R32
R33
R34
R35
R36
R38
R40
R45
R46
R48
R50
R51
R52

/lustre/gamma/DESPEC_S496_FILES/AIDA
S496
R1
R2
R3
R4
R5
R6
R7
R9
R11
R13
R14
R15
R16
R17
R20
R21
R22
R23
R25
R27
R28
R29
R30
R33
R34
R35
R36
R38
R39
R40
R42
R43
R46
R50
R56
R57
R58
  372   Mon Jun 21 10:21:50 2021 NHMonday 21 June
11.21 - DAQ continues OK (no storage mode)

11.21 - System checks ok *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8963 : 	 31  
aida02 fault 	 0x9b36 : 	 0x9b55 : 	 31  
aida03 fault 	 0xf26b : 	 0xf28a : 	 31  
aida04 fault 	 0x57a0 : 	 0x57bf : 	 31  
aida05 fault 	 0x4d3c : 	 0x4d95 : 	 89  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16


			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x3 : 	 3  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x9560 : 	 38240  
aida13 fault 	 0x0 : 	 0x15b9 : 	 5561  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     33     12     14      2      4      4      2      2      2      4      6   : 37636
aida02 :     40      7     15      1      3      4      3      2      2      4      6   : 37800
aida03 :     39      7      9      6      5      3      1      3      2      4      6   : 37860
aida04 :     27      7      5      3      3      4      2      3      2      4      6   : 37908
aida05 :     33      5      5      3      6      4      3      3      2      4      6   : 38364
aida06 :     45     12      3      4      4      2      2      3      4      3      6   : 37828
aida07 :     31     11      5      4      4      2      3      2      2      4      6   : 37540
aida08 :     31      6      8      5      3      4      3      2      2      4      6   : 37772
aida09 :     45      4      9      3      2      3      3      2      2      4      6   : 37572
aida10 :     37     11     10      2      3      4      3      3      3      3      6   : 37260
aida11 :     40      9     11      3      3      4      3      3      1      4      6   : 37304
aida12 :     44     11      6      5      3      4      3      3      1      4      6   : 37320
aida13 :     41     11      9      3      5      3      3      3      1      4      6   : 37292
aida14 :     31     12      4      6      5      3      3      3      3      3      6   : 37276
aida15 :     17      9      7      1      2      2      3      2      1      4      7   : 40348
aida16 :     41     12      5      4      5      3      3      3      1      4      6   : 37268


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


11.23 - figures
    Grafana temp & leakage - attachment 1
    Weather cooler this week in Germany, expect S4 to cool down eventually
    Lost activity monitor - attachment 2
    1.8.W spectra - 20us FSR - attachments 3 & 4
    aida05 looks very noisy (?) 
    ADC data items - attachment 5
    FEE64 temperatures OK - attachment 6
    DSSSD bias & leakage currents - attachment 7
    Merger/Tape Server/Merger statistics - attachment 8
    Looks some new time seq error since Thursday

Experiment is runnign in S4 so no access possible this week

Data is being archived to lustre (GSI PB file storage) 
Attachment 1: firefox_2021-06-21_11-27-00.png
firefox_2021-06-21_11-27-00.png
Attachment 2: AnyDesk_2021-06-21_11-24-28.png
AnyDesk_2021-06-21_11-24-28.png
Attachment 3: AnyDesk_2021-06-21_11-25-44.png
AnyDesk_2021-06-21_11-25-44.png
Attachment 4: AnyDesk_2021-06-21_11-26-07.png
AnyDesk_2021-06-21_11-26-07.png
Attachment 5: AnyDesk_2021-06-21_11-24-15.png
AnyDesk_2021-06-21_11-24-15.png
Attachment 6: AnyDesk_2021-06-21_11-24-43.png
AnyDesk_2021-06-21_11-24-43.png
Attachment 7: AnyDesk_2021-06-21_11-25-04.png
AnyDesk_2021-06-21_11-25-04.png
Attachment 8: AnyDesk_2021-06-21_11-25-15.png
AnyDesk_2021-06-21_11-25-15.png
  371   Thu Jun 17 11:14:04 2021 TDThursday 17 June
12.15 DAQ continues OK - no storage mode

12.15 System checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8963 : 	 31  
aida02 fault 	 0x9b36 : 	 0x9b55 : 	 31  
aida03 fault 	 0xf26b : 	 0xf28a : 	 31  
aida04 fault 	 0x57a0 : 	 0x57bf : 	 31  
aida05 fault 	 0x4d3c : 	 0x4d55 : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x3e03 : 	 15875  
aida13 fault 	 0x0 : 	 0x32d6 : 	 13014  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     31     11      9      4      4      4      2      2      2      4      6   : 37604
aida02 :     30     10     10      3      3      4      3      2      2      4      6   : 37768
aida03 :     35     11      8      6      5      3      2      2      3      3      6   : 36580
aida04 :     33      9      4      4      3      4      2      3      3      3      6   : 36940
aida05 :     41      9      7      3      4      4      2      2      2      4      6   : 37564
aida06 :     38      6      5      3      2      3      2      3      4      3      6   : 37752
aida07 :     25     12      6      5      4      2      3      2      2      4      6   : 37572
aida08 :     32      7     10      5      4      3      3      2      2      4      6   : 37752
aida09 :     41      8      6      4      2      3      3      2      2      4      6   : 37572
aida10 :     31     10     11      5      5      3      3      3      3      3      6   : 37340
aida11 :     29      7     12      3      4      4      3      3      1      4      6   : 37324
aida12 :     42      8      7      5      3      4      3      3      1      4      6   : 37304
aida13 :     43     10      4      5      5      3      3      3      1      4      6   : 37276
aida14 :     40     15      4      5      4      4      3      3      3      3      6   : 37368
aida15 :     13      5      8      2      1      2      3      2      1      4      7   : 40284
aida16 :     39      9      6      6      5      3      3      3      1      4      6   : 37316

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

12.15 Hot HEC channels - ASIC check

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents - attachment 7
       significant increases of leakage current to c. 12uA continues - temperature c. 30 deg C
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-17_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-17_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-17_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-17_Statistics_aidas-gsi.png
Screenshot_2021-06-17_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 1008.png
1008.png
Attachment 8: 1009.png
1009.png
  370   Wed Jun 16 18:05:03 2021 TDWednesday 16 June
19.04 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8960 : 	 28  
aida02 fault 	 0x9b36 : 	 0x9b52 : 	 28  
aida03 fault 	 0xf26b : 	 0xf287 : 	 28  
aida04 fault 	 0x57a0 : 	 0x57bc : 	 28  
aida05 fault 	 0x4d3c : 	 0x4d55 : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16

			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x3e03 : 	 15875  
aida13 fault 	 0x0 : 	 0x32d4 : 	 13012  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     53     10      4      4      4      4      2      2      3      3      6   : 36580
aida02 :     28      7     10      4      3      4      3      2      2      4      6   : 37768
aida03 :     41     10      7      6      5      3      1      3      3      3      6   : 36836
aida04 :     45      7      2      4      3      4      2      3      3      3      6   : 36940
aida05 :     35      8      8      4      4      4      2      2      2      4      6   : 37580
aida06 :     45      6      6      3      3      3      2      3      4      3      6   : 37860
aida07 :     37     10      6      4      5      2      3      2      2      4      6   : 37636
aida08 :     32      5     10      4      4      4      3      2      2      4      6   : 37832
aida09 :     24      5      8      7      1      3      3      2      2      4      6   : 37544
aida10 :     29      9     12      5      4      3      3      3      3      3      6   : 37276
aida11 :     33      7      7      6      3      4      3      3      1      4      6   : 37292
aida12 :     36     11      6      6      3      4      3      3      1      4      6   : 37320
aida13 :     37      7      6      6      5      3      3      3      1      4      6   : 37292
aida14 :     25      9      7      6      5      3      3      3      3      3      6   : 37276
aida15 :      3      3      4      3      0      3      3      2      1      4      7   : 40260
aida16 :     35      7      6      6      4      4      3      3      1      4      6   : 37348

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19.07 Hot HEC channels - ASIC check

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents - attachment 7
       significant increases of leakage current to c. 12uA - temperature c. 30 deg C
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-16_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-16_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-06-16_Statistics_aidas-gsi(1).png
Screenshot_2021-06-16_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-16_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-16_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-16_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 1005.png
1005.png
Attachment 7: 1006.png
1006.png
  369   Tue Jun 15 12:35:48 2021 TDTuesday 15 June
13.35 DAQ continues OK - no storage mode

      All system wide checks OK *except*
	
		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8949 : 	 5  
aida02 fault 	 0x9b36 : 	 0x9b3b : 	 5  
aida03 fault 	 0xf26b : 	 0xf270 : 	 5  
aida04 fault 	 0x57a0 : 	 0x57a5 : 	 5  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
aida15 fault 	 0x4e4 : 	 0x1267 : 	 3459  
White Rabbit error counter test result: Passed 7, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x3e01 : 	 15873  
aida13 fault 	 0x0 : 	 0x32d2 : 	 13010  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     37     10      8      5      4      4      2      2      2      4      6   : 37636
aida02 :     37     10     10      3      4      4      3      2      2      4      6   : 37860
aida03 :     41      8      7      7      5      3      1      3      2      4      6   : 37876
aida04 :     33      9      4      4      2      5      2      3      3      3      6   : 37004
aida05 :     37      9      9      3      5      4      2      2      2      4      6   : 37644
aida06 :     39      9      6      5      4      2      2      3      4      3      6   : 37860
aida07 :     29     12      6      5      4      2      3      2      2      4      6   : 37588
aida08 :     25      7      9      5      4      3      3      2      2      4      6   : 37708
aida09 :     34      5      9      6      2      3      3      2      2      4      6   : 37632
aida10 :     34     12     11      5      4      3      3      3      3      3      6   : 37304
aida11 :     38      8      7      6      3      4      3      3      1      4      6   : 37320
aida12 :     34     10      8      6      3      4      3      3      1      4      6   : 37336
aida13 :     33     11      5      6      4      4      3      3      1      4      6   : 37356
aida14 :     31     10      5      6      5      3      3      3      3      3      6   : 37276
aida15 :     14      6      7      4      1      2      3      2      1      4      7   : 40344
aida16 :     33      8      6      7      5      3      3      3      1      4      6   : 37316

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

13.40 All histograms, stats & merger stats zero'd
Attachment 1: Screenshot_2021-06-15_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-15_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-15_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-15_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-15_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-15_Statistics_aidas-gsi.png
Screenshot_2021-06-15_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 1002.png
1002.png
Attachment 8: 1003.png
1003.png
  368   Tue Jun 15 12:33:46 2021 TDMonday 14 June
  367   Sun Jun 13 17:29:04 2021 TDSunday 13 June
18.15 DAQ continues file S496/R32_946
      Hot HEC channels multiple FEE64s from c. 06.00 this morning
      ASIC check fixes

      Out of disk space on /TapeData filesystem - switch to no storage mode

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8949 : 	 5  
aida02 fault 	 0x9b36 : 	 0x9b3b : 	 5  
aida03 fault 	 0xf26b : 	 0xf270 : 	 5  
aida04 fault 	 0x57a0 : 	 0x57a5 : 	 5  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
aida15 fault 	 0x4e4 : 	 0x1267 : 	 3459  
White Rabbit error counter test result: Passed 7, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x3e01 : 	 15873  
aida13 fault 	 0x0 : 	 0x32d2 : 	 13010  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26     16      4      6      4      4      2      2      2      4      6   : 37608
aida02 :     29     11      9      3      3      4      3      2      2      4      6   : 37756
aida03 :     32      9      6      8      5      3      1      3      3      3      6   : 36840
aida04 :     32      8      3      4      3      4      2      3      3      3      6   : 36912
aida05 :     36      8      6      4      5      4      2      2      2      4      6   : 37616
aida06 :     25     13      4      3      2      3      2      3      4      3      6   : 37740
aida07 :     28     13      4      5      5      2      3      2      2      4      6   : 37624
aida08 :     29      5      9      4      3      4      3      2      2      4      6   : 37740
aida09 :     41      8      4      7      1      3      3      2      2      4      6   : 37572
aida10 :     45     11      7      5      5      3      3      3      3      3      6   : 37340
aida11 :     33     13      5      6      3      4      3      3      1      4      6   : 37308
aida12 :     39      8      6      6      4      4      3      3      1      4      6   : 37372
aida13 :     28     14      3      6      4      4      3      3      1      4      6   : 37328
aida14 :     30     11      3      6      4      4      3      3      3      3      6   : 37312
aida15 :      3      3      4      3      0      3      3      2      1      4      7   : 40260
aida16 :     28      9      5      6      5      3      3      3      1      4      6   : 37256

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart
Attachment 1: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi(1).png
Attachment 2: Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-13_Spectrum_Browser_aidas-gsi.png
Attachment 3: Screenshot_2021-06-13_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-13_AIDA_Alerting_-_Grafana.png
Attachment 4: Screenshot_2021-06-13_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-13_Lost_activity_monitor_aidas-gsi.png
Attachment 5: Screenshot_2021-06-13_Statistics_aidas-gsi.png
Screenshot_2021-06-13_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-13_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-13_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 1000.png
1000.png
Attachment 8: 1001.png
1001.png
  366   Sat Jun 12 14:53:56 2021 TDaida15 - system console output - 15.33 Sat 12 June reboot
04/11:03:40|get_ASICBlk : Blk=1 : bytes = 196096 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 80500240 : 1 => 0F07C000 
04/11:03:41|Last Data : Databuffer  273286648 : 0 => 80500240 : 1 => 0AFF8000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:41| 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 | 00000000 : 00000000 
04/11:03:42|Machine check in kernel mode.
12/15:30:08|I-Cache Parity Error
12/15:30:08|Oops: Machine check, sig: 7 [#1]
12/15:30:08|PREEMPT Xilinx Virtex440
12/15:30:08|Modules linked in: aidamem xdriver xh_spidev_register
12/15:30:08|NIP: c0055c54 LR: c00558fc CTR: c000ae5c
12/15:30:08|REGS: c6ff9f10 TRAP: 0214   Not tainted  (2.6.31)
12/15:30:08|MSR: 00021000 <ME,CE>  CR: 82000028  XER: 20000000
12/15:30:08|TASK = c6ac44e0[375] 'AidaExecV9' THREAD: c6338000
12/15:30:08|GPR00: 00000000 c6339cd0 c6ac44e0 ffffffff ffe79600 ffffffff ffe79600 ffffffff 
12/15:30:08|GPR08: 00000000 0000001f ffffffe0 00000000 00000000 1005520c 1dcd6500 00000000 
12/15:30:08|GPR16: 00000000 1009df98 100f0000 c0390000 000007a1 7fe9e580 ffffffff ffffffff 
12/15:30:08|GPR24: c0390000 00000001 00000000 001366ca c036f0d0 ffffffe1 ffffffff ffec9936 
12/15:30:08|NIP [c0055c54] update_wall_time+0x40c/0xd3c
12/15:30:09|LR [c00558fc] update_wall_time+0xb4/0xd3c
12/15:30:09|Call Trace:
12/15:30:09|[c6339cd0] [c0055f44] update_wall_time+0x6fc/0xd3c (unreliable)
12/15:30:09|[c6339d90] [c0040b84] do_timer+0x38/0x4c
12/15:30:09|[c6339da0] [c005a088] tick_periodic+0xbc/0xe8
12/15:30:09|[c6339db0] [c005a0d4] tick_handle_periodic+0x20/0x120
12/15:30:09|[c6339df0] [c000af70] timer_interrupt+0xa4/0x10c
12/15:30:09|[c6339e10] [c000e9c4] ret_from_except+0x0/0x18
12/15:30:09|[c6339ed0] [d10751dc] aidamem_read+0x28/0x108 [aidamem]
12/15:30:09|[c6339ef0] [c0097e80] vfs_read+0xb4/0x188
12/15:30:09|[c6339f10] [c00982f8] sys_read+0x4c/0x90
12/15:30:09|[c6339f40] [c000e364] ret_from_syscall+0x0/0x3c
12/15:30:09|Instruction dump:
12/15:30:09|39600000 354bffe0 418008b0 7cc35030 38800000 2f8a0000 419c08bc 7ffe5030 
12/15:30:09|3aa00000 93c10048 92a1004c 83c10048 <83e1004c> 7d095830 4800056c 419e06bc 
12/15:30:09|Kernel panic - not syncing: Fatal exception in interrupt
12/15:30:09|Call Trace:
12/15:30:09|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
12/15:33:10|
Found 0 errors checking kernel image
12/15:33:11|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
12/15:33:11|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
12/15:33:11|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
12/15:33:11|SPIconfig done now to set up the LMK3200 registers
12/15:33:12|LMK #0 : regInit[0]=0x80000000
12/15:33:12|LMK #0 : regInit[1]=0x10070600
12/15:33:12|LMK #0 : regInit[2]=0x60601
12/15:33:12|LMK #0 : regInit[3]=0x60602
12/15:33:12|LMK #0 : regInit[4]=0x60603
12/15:33:12|LMK #0 : regInit[5]=0x70624
12/15:33:12|LMK #0 : regInit[6]=0x70605
12/15:33:12|LMK #0 : regInit[7]=0x70606
12/15:33:12|LMK #0 : regInit[8]=0x70627
12/15:33:12|LMK #0 : regInit[9]=0x10000908
12/15:33:12|LMK #0 : regInit[10]=0xA0022A09
12/15:33:12|LMK #0 : regInit[11]=0x82800B
12/15:33:12|LMK #0 : regInit[12]=0x28C800D
12/15:33:12|LMK #0 : regInit[13]=0x830020E
12/15:33:12|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
12/15:33:12|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
12/15:33:12|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
12/15:33:13|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
ADCs calibrated
12/15:33:13|
Control 32(0x81400400)=0x186
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
12/15:33:13|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
12/15:33:13|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
12/15:33:16|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 
12/15:33:23|Using Xilinx Virtex440 machine description
12/15:33:23|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
12/15:33:23|Zone PFN ranges:
12/15:33:23|  DMA      0x00000000 -> 0x00007000
12/15:33:23|  Normal   0x00007000 -> 0x00007000
12/15:33:23|Movable zone start PFN for each node
12/15:33:23|early_node_map[1] active PFN ranges
12/15:33:23|    0: 0x00000000 -> 0x00007000
12/15:33:23|MMU: Allocated 1088 bytes of context maps for 255 contexts
12/15:33:23|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
12/15:33:24|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
12/15:33:24|PID hash table entries: 512 (order: 9, 2048 bytes)
12/15:33:24|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
12/15:33:24|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
12/15:33:24|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
12/15:33:24|Kernel virtual memory layout:
12/15:33:24|  * 0xffffe000..0xfffff000  : fixmap
12/15:33:24|  * 0xfde00000..0xfe000000  : consistent mem
12/15:33:24|  * 0xfde00000..0xfde00000  : early ioremap
12/15:33:24|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
12/15:33:24|NR_IRQS:512
12/15:33:24|clocksource: timebase mult[a00000] shift[22] registered
12/15:33:24|Console: colour dummy device 80x25
12/15:33:24|Mount-cache hash table entries: 512
12/15:33:24|NET: Registered protocol family 16
12/15:33:24|PCI: Probing PCI hardware
12/15:33:24|bio: create slab <bio-0> at 0
12/15:33:24|NET: Registered protocol family 2
12/15:33:24|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
12/15:33:24|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
12/15:33:25|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
12/15:33:25|TCP: Hash tables configured (established 4096 bind 4096)
12/15:33:25|TCP reno registered
12/15:33:25|NET: Registered protocol family 1
12/15:33:25|ROMFS MTD (C) 2007 Red Hat, Inc.
12/15:33:25|msgmni has been set to 214
12/15:33:25|io scheduler noop registered
12/15:33:25|io scheduler anticipatory registered
12/15:33:25|io scheduler deadline registered
12/15:33:25|io scheduler cfq registered (default)
12/15:33:25|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
12/15:33:25|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
12/15:33:25|console [ttyS0] enabled
12/15:33:25|brd: module loaded
12/15:33:25|loop: module loaded
12/15:33:25|Device Tree Probing 'ethernet'
12/15:33:25|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
12/15:33:25|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
12/15:33:25|XLlTemac: DCR address: 0x80
12/15:33:25|XLlTemac: buffer descriptor size: 32768 (0x8000)
12/15:33:25|XLlTemac: Allocating DMA descriptors with kmalloc
12/15:33:25|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
12/15:33:26|XTemac: PHY detected at address 7.
12/15:33:26|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
12/15:33:26|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26| Intel/Sharp Extended Query Table at 0x010A
12/15:33:26|Using buffer write method
12/15:33:26|cfi_cmdset_0001: Erase suspend on write enabled
12/15:33:26|cmdlinepart partition parsing not available
12/15:33:26|RedBoot partition parsing not available
  365   Sat Jun 12 13:56:00 2021 TDSaturday 12 June
14.11 DAQ continues file S496/R32_697
      High data rate from c. 06.00 this morning
      ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8948 : 	 4  
aida02 fault 	 0x9b36 : 	 0x9b3a : 	 4  
aida03 fault 	 0xf26b : 	 0xf26f : 	 4  
aida04 fault 	 0x57a0 : 	 0x57a4 : 	 4  
aida05 fault 	 0x4d3c : 	 0x4d3e : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640e : 	 2  
aida07 fault 	 0x255e : 	 0x2560 : 	 2  
aida08 fault 	 0xffbd : 	 0xffbf : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x21be : 	 8638  
aida13 fault 	 0x0 : 	 0x2294 : 	 8852  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     33     10      9      2      5      4      2      2      2      4      6   : 37604
aida02 :     46     12      7      2      4      4      3      2      2      4      6   : 37832
aida03 :     34     10      9      5      4      3      1      3      3      3      6   : 36744
aida04 :     35     12     10      1      3      4      2      3      3      3      6   : 36972
aida05 :     37      9      8      3      4      4      2      2      2      4      6   : 37564
aida06 :     34     10      5      5      4      2      2      3      2      4      6   : 37832
aida07 :     31     11      1      7      4      2      3      2      2      4      6   : 37572
aida08 :     40      7      4      5      3      4      3      2      2      4      6   : 37752
aida09 :     39      9      8      4      1      3      3      2      2      4      6   : 37540
aida10 :     43     12      8      4      4      3      3      3      1      4      6   : 37260
aida11 :     34     14     10      5      3      4      3      3      1      4      6   : 37368
aida12 :     46     12      3      4      4      4      3      3      1      4      6   : 37320
aida13 :     41     11      5      4      5      3      3      3      1      4      6   : 37260
aida14 :     39     10      2      6      4      4      3      3      1      4      6   : 37324
aida15 :     19     12      4      3      2      2      3      2      2      4      6   : 37324
aida16 :     45     10      6      4      5      3      3      3      1      4      6   : 37284

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

15.05 analysis of file S496/R32_698 - attachment 9
      max dead time 0.05% aida04

15.33 aida15 rebooted - see https://elog.ph.ed.ac.uk/DESPEC/365

      successful DAQ stop
      aida15 DAQ reset, setup, go 
      successful DAQ go 
Attachment 1: Screenshot_2021-06-12_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-12_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-12_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-12_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-12_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-12_Statistics_aidas-gsi.png
Screenshot_2021-06-12_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 998.png
998.png
Attachment 8: 999.png
999.png
Attachment 9: R32_698
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   44970784 (   15685.0 Hz)
        Other data format:  216949222 (   75668.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         62 (       0.0 Hz)
                               RESUME:         62 (       0.0 Hz)
                              SYNC100:      42680 (      14.9 Hz)
                              WR48-63:      42680 (      14.9 Hz)
                           FEE64 disc:     985985 (     343.9 Hz)
                             MBS info:  215877753 (   75294.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2867.115 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2299.418
  2                0.673                0.000
  3                1.368                0.000
  4                0.000                0.000
  5                0.000             2495.101
  6                0.000                0.000
  7                0.000             2566.695
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        467          2          0          0          0          0          1          1          0          0          0          0
  1      72420         30          0          0          0          0         15         15          0          0          0          0
  2        935  107990737          0          0         33         33      19385      19385          0  107951901          0          0
  3   38882631  108955999          0          0         29         29      22052      22052     985985  107925852          0          0
  4        496          0          0          0          0          0          0          0          0          0          0          0
  5    3224207       1336          0          0          0          0        668        668          0          0          0          0
  6       1520          0          0          0          0          0          0          0          0          0          0          0
  7    2784837       1118          0          0          0          0        559        559          0          0          0          0
  8        473          0          0          0          0          0          0          0          0          0          0          0
  9        173          0          0          0          0          0          0          0          0          0          0          0
 10        554          0          0          0          0          0          0          0          0          0          0          0
 11        317          0          0          0          0          0          0          0          0          0          0          0
 12        752          0          0          0          0          0          0          0          0          0          0          0
 13        204          0          0          0          0          0          0          0          0          0          0          0
 14        302          0          0          0          0          0          0          0          0          0          0          0
 15        496          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.945s ( 1068.615 blocks/s,  66.788 Mb/s)
  363   Thu Jun 10 11:32:27 2021 TDThursday 10 June
12.26 DAQ continues file S496/R32_428
      High data rate from c. 06.00 this morning
      ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     11      6      3      4      4      2      2      2      4      6   : 37540
aida02 :     30      8      7      3      4      4      3      2      2      4      6   : 37768
aida03 :     38      8     13      3      4      3      1      3      3      3      6   : 36744
aida04 :     37      9      7      2      3      4      2      3      3      3      6   : 36940
aida05 :     29      5      9      3      3      3      1      3      2      4      6   : 37580
aida06 :     38      6      2      3      3      3      2      3      2      4      6   : 37768
aida07 :     31      9      2      7      4      2      3      2      2      4      6   : 37572
aida08 :     34      6      3      5      4      4      3      2      2      4      6   : 37768
aida09 :     29      2     12      5      1      3      3      2      2      4      6   : 37540
aida10 :     24      8     15      3      5      3      3      3      1      4      6   : 37296
aida11 :     35      4     15      3      3      4      3      3      1      4      6   : 37308
aida12 :     36      5      3      5      6      4      3      3      1      4      6   : 37384
aida13 :     31     10      4      3      6      4      3      3      1      4      6   : 37356
aida14 :     37      7      1      4      5      4      3      3      1      4      6   : 37276
aida15 :     34     10      6      3      4      1      2      4      2      4      6   : 38168
aida16 :     31      9      3      3      5      4      3      3      1      4      6   : 37268

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 202

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

15.07 analysis of file S496/R32_431 - attachment 9
      max dead time aida04 0.33%
Attachment 1: Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-10_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-10_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-10_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-10_Statistics_aidas-gsi.png
Screenshot_2021-06-10_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-10_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 996.png
996.png
Attachment 8: 997.png
997.png
Attachment 9: R32_431
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   54180182 (   19758.0 Hz)
        Other data format:  207739819 (   75756.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        105 (       0.0 Hz)
                               RESUME:        105 (       0.0 Hz)
                              SYNC100:      42213 (      15.4 Hz)
                              WR48-63:      42213 (      15.4 Hz)
                           FEE64 disc:    1515114 (     552.5 Hz)
                             MBS info:  206140069 (   75173.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          6 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2742.196 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2662.854
  2                0.384                0.000
  3                9.009                0.000
  4                0.000                0.000
  5                0.000             1805.119
  6                0.000                0.000
  7                0.000             1797.783
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        559          0          0          0          0          0          0          0          0          0          0          0
  1     413404        177          0          0          0          0         84         84          9          0          0          0
  2       1153  103268358          0          0         19         19      17963      17963          0  103232394          0          0
  3   42054837  104466440          0          0         86         86      21757      21757    1515079  102907675          0          0
  4        515          0          0          0          0          0          0          0          0          0          0          0
  5    5561752       2359          0          0          0          0       1175       1175          9          0          0          0
  6       1738          2          0          0          0          0          1          1          0          0          0          0
  7    6142236       2472          0          0          0          0       1232       1232          8          0          0          0
  8        565          1          0          0          0          0          0          0          1          0          0          2
  9        190          0          0          0          0          0          0          0          0          0          0          0
 10        694          0          0          0          0          0          0          0          0          0          0          0
 11        417          6          0          0          0          0          0          0          6          0          0          4
 12        967          4          0          0          0          0          1          1          2          0          0          0
 13        244          0          0          0          0          0          0          0          0          0          0          0
 14        285          0          0          0          0          0          0          0          0          0          0          0
 15        626          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.160s ( 1097.388 blocks/s,  68.587 Mb/s)
  362   Wed Jun 9 19:17:21 2021 TDFEE64 system console logs
Attachment 1: logs-090621.tar.gz
  361   Wed Jun 9 11:27:47 2021 TDWednesday 9 June
12.30 DAQ continues OK file S496/R32_373
 
      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5     12      7      2      4      2      2      1      4      7   : 40572
aida02 :     11      6      8      6      1      3      3      2      1      4      7   : 40540
aida03 :     26      5     11      5      1      2      2      3      2      3      7   : 39712
aida04 :     18      7      8      3      2      3      2      3      2      3      7   : 39776
aida05 :     11      5      9      4      1      2      2      3      1      4      7   : 40612
aida06 :     17      3      6      5      5      2      1      3      1      4      7   : 40604
aida07 :      9      3      4      9      1      3      3      2      1      4      7   : 40540
aida08 :     11      8      5      5      3      3      3      2      1      4      7   : 40604
aida09 :     13      5      8      5      2      3      3      2      1      4      7   : 40572
aida10 :     13      8     13      5      1      2      2      2      1      4      7   : 40228
aida11 :     11      5     14      6      3      3      3      3      2      3      7   : 40244
aida12 :     10      9      5      5      3      3      2      2      1      4      7   : 40352
aida13 :     15      7      7      4      3      4      3      3      2      3      7   : 40228
aida14 :      9      6      7      6      3      4      3      3      2      3      7   : 40260
aida15 :     14      3      8      5      1      2      2      4      1      4      7   : 41136
aida16 :     15      4      9      4      4      4      3      3      2      3      7   : 40300

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      1.8.L specttra - attachments 1 & 2
      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3
      Lost activity monitor - attachment 4
      1.8.W spectra - 20us FSR - attachments 5 & 6
      ADC data items - attachment 7
      FEE64 temperatures OK - attachment 8
      DSSSD bias & leakage currents OK - attachment 9
      Merger/Tape Server/Merger statistics - attachment 10
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_372 - attachment 11
       max deadtime 0.07% aida04
Attachment 1: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(3).png
Attachment 2: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(2).png
Attachment 3: Screenshot_2021-06-09_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-09_AIDA_Alerting_-_Grafana.png
Attachment 4: Screenshot_2021-06-09_Lost_activity_monitor_aidas-gsi(1).png
Screenshot_2021-06-09_Lost_activity_monitor_aidas-gsi(1).png
Attachment 5: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-09_Spectrum_Browser_aidas-gsi.png
Attachment 7: Screenshot_2021-06-09_Statistics_aidas-gsi.png
Screenshot_2021-06-09_Statistics_aidas-gsi.png
Attachment 8: Screenshot_2021-06-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: 991.png
991.png
Attachment 10: 992.png
992.png
Attachment 11: R32_372
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   65571173 (   25420.1 Hz)
        Other data format:  196348830 (   76118.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         52 (       0.0 Hz)
                               RESUME:         52 (       0.0 Hz)
                              SYNC100:      41608 (      16.1 Hz)
                              WR48-63:      41608 (      16.1 Hz)
                           FEE64 disc:    2050316 (     794.8 Hz)
                             MBS info:  194215194 (   75291.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        284 (       0.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2579.504 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.015             1853.986
  2                0.548                0.000
  3                1.886                0.000
  4                0.000                0.000
  5                0.000             1457.725
  6                0.000                0.000
  7                0.000             1391.289
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000             1323.904
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000              836.274
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0       6664        532          0          0          0          0          1          1        530          0          0          3
  1    4648488      85466          0          0          1          1        947        947      83570          0          0         92
  2       5675   97165745          0          0         27         27      16487      16487          7   97132710          0          2
  3   47009038   98898439          0          0         24         24      21346      21346    1773239   97082460          0         42
  4       3608          4          0          0          0          0          0          0          4          0          0          1
  5    6949043      95991          0          0          0          0       1383       1383      93225          0          0         60
  6       5001         30          0          0          0          0          1          1          4         24          0          0
  7    6897601     102341          0          0          0          0       1436       1436      99469          0          0         74
  8       1797          1          0          0          0          0          0          0          1          0          0          1
  9       2911          4          0          0          0          0          1          1          2          0          0          3
 10       8722        137          0          0          0          0          1          1        135          0          0          0
 11       5985         36          0          0          0          0          2          2         32          0          0          4
 12       7238         34          0          0          0          0          1          1         32          0          0          0
 13       7345         52          0          0          0          0          0          0         52          0          0          0
 14       4898          4          0          0          0          0          0          0          4          0          0          1
 15       7159         14          0          0          0          0          2          2         10          0          0          1
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   30.055s ( 1064.726 blocks/s,  66.545 Mb/s)
  360   Tue Jun 8 10:51:55 2021 TDTuesday 8 June
11.52 DAQ continues file S496/R32_340
      High data ratefrom c. 18.00 yesterday evening to c. 06.00 this morning

11.53 ASIC check 

      System wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8947 : 	 3  
aida02 fault 	 0x9b36 : 	 0x9b39 : 	 3  
aida03 fault 	 0xf26b : 	 0xf26e : 	 3  
aida04 fault 	 0x57a0 : 	 0x57a3 : 	 3  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      9      3     12      8      3      3      2      2      1      4      7   : 40508
aida02 :      5      5     10      6      1      3      3      2      1      4      7   : 40540
aida03 :     25      7     12      5      1      2      2      3      2      3      7   : 39740
aida04 :     19      6     11      5      1      3      2      3      2      3      7   : 39820
aida05 :      9      6      7      5      0      2      2      3      1      4      7   : 40548
aida06 :     10      8      7      4      3      3      1      3      1      4      7   : 40600
aida07 :      9      3      4      8      1      3      3      2      1      4      7   : 40508
aida08 :     12      5      8      5      3      3      3      2      1      4      7   : 40632
aida09 :     15      4      8      5      1      3      3      2      1      4      7   : 40508
aida10 :     23      5     12      5      1      2      2      2      1      4      7   : 40228
aida11 :     15      9     11      6      2      4      3      3      2      3      7   : 40308
aida12 :     14      9      4      5      3      3      2      2      1      4      7   : 40352
aida13 :     15      9      6      3      4      4      3      3      2      3      7   : 40260
aida14 :     11      5      7      6      4      4      3      3      2      3      7   : 40324
aida15 :     16      8      7      4      1      2      2      4      1      4      7   : 41136
aida16 :     15      8      7      4      3      4      3      3      2      3      7   : 40236

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_339 - attachment 9
       max deadtime 0.24% aida04
Attachment 1: Screenshot_2021-06-08_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-08_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-08_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-08_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-08_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-08_Statistics_aidas-gsi.png
Screenshot_2021-06-08_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-08_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-08_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 990.png
990.png
Attachment 8: 991.png
991.png
Attachment 9: R32_339
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   52833088 (   19139.6 Hz)
        Other data format:  209086916 (   75745.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        101 (       0.0 Hz)
                               RESUME:        101 (       0.0 Hz)
                              SYNC100:      42281 (      15.3 Hz)
                              WR48-63:      42281 (      15.3 Hz)
                           FEE64 disc:    1254898 (     454.6 Hz)
                             MBS info:  207747254 (   75259.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2760.405 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2672.516
  2                0.610                0.000
  3                6.711                0.000
  4                0.000                0.000
  5                0.000             2111.274
  6                0.000                0.000
  7                0.000             1910.870
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        589          0          0          0          0          0          0          0          0          0          0          0
  1     221336         98          0          0          0          0         49         49          0          0          0          0
  2       1118  104025022          0          0         30         30      18193      18193          0  103988576          0          0
  3   42352603  105057740          0          0         71         71      22011      22011    1254898  103758678          0          0
  4        548          0          0          0          0          0          0          0          0          0          0          0
  5    4562177       1856          0          0          0          0        928        928          0          0          0          0
  6       1810          0          0          0          0          0          0          0          0          0          0          0
  7    5688898       2196          0          0          0          0       1098       1098          0          0          0          0
  8        593          0          0          0          0          0          0          0          0          0          0          0
  9        215          2          0          0          0          0          1          1          0          0          0          0
 10        675          2          0          0          0          0          1          1          0          0          0          0
 11        387          0          0          0          0          0          0          0          0          0          0          0
 12        964          0          0          0          0          0          0          0          0          0          0          0
 13        267          0          0          0          0          0          0          0          0          0          0          0
 14        369          0          0          0          0          0          0          0          0          0          0          0
 15        539          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   28.656s ( 1116.685 blocks/s,  69.793 Mb/s)
  359   Mon Jun 7 10:33:48 2021 TDMonday 7 June
11.30 DAQ continues OK - file S496/R32_144
      alpha background

      All system wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      9      5     11      6      3      4      2      2      1      4      7   : 40572
aida02 :      1      3      6      7      2      3      3      2      1      4      7   : 40540
aida03 :     21      7      7      8      2      4      1      3      2      3      7   : 39804
aida04 :     13      7      5      6      1      3      2      3      2      3      7   : 39740
aida05 :      2      8      6      5      1      2      2      3      1      4      7   : 40584
aida06 :     10      4      5      5      4      3      1      3      1      4      7   : 40632
aida07 :      9      3      4      7      3      3      3      2      1      4      7   : 40604
aida08 :     11      2      6      4      4      3      3      2      1      4      7   : 40604
aida09 :     10      2      5      4      2      2      2      3      1      4      7   : 40584
aida10 :     14      5      8      6      2      1      1      3      1      4      7   : 40352
aida11 :      8      6      9      5      4      4      3      3      2      3      7   : 40320
aida12 :     10      5      3      7      3      3      2      2      1      4      7   : 40352
aida13 :      7      5      4      6      5      4      3      3      2      3      7   : 40324
aida14 :      7      5      4      8      4      4      3      3      2      3      7   : 40324
aida15 :      8      4      7      4      2      2      2      4      1      4      7   : 41136
aida16 :     13      5      7      6      3      4      3      3      2      3      7   : 40268

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

11.35 analysis of file S496/R32_143 - attachment 9
       max deadtime 0.29% aida03
Attachment 1: Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-07_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-07_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-07_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-07_Statistics_aidas-gsi.png
Screenshot_2021-06-07_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-07_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 986.png
986.png
Attachment 8: 987.png
987.png
Attachment 9: R32_143
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   57707988 (   21988.9 Hz)
        Other data format:  204212013 (   77812.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         85 (       0.0 Hz)
                               RESUME:         85 (       0.0 Hz)
                              SYNC100:      41772 (      15.9 Hz)
                              WR48-63:      41772 (      15.9 Hz)
                           FEE64 disc:    1109863 (     422.9 Hz)
                             MBS info:  203018436 (   77357.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2624.418 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2432.488
  2                0.665                0.000
  3                7.525                0.000
  4                0.000                0.000
  5                0.000             1464.970
  6                0.000                0.000
  7                0.000             1288.051
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        112          0          0          0          0          0          0          0          0          0          0          0
  1    1940572        788          0          0          0          0        394        394          0          0          0          0
  2        442  101676817          0          0         30         30      17294      17294          0  101642169          0          0
  3   40857770  102528388          0          0         55         55      21074      21074    1109863  101376267          0          0
  4        129          0          0          0          0          0          0          0          0          0          0          0
  5    7006930       2878          0          0          0          0       1439       1439          0          0          0          0
  6        548          0          0          0          0          0          0          0          0          0          0          0
  7    7898329       3142          0          0          0          0       1571       1571          0          0          0          0
  8        285          0          0          0          0          0          0          0          0          0          0          0
  9         71          0          0          0          0          0          0          0          0          0          0          0
 10       1135          0          0          0          0          0          0          0          0          0          0          0
 11        122          0          0          0          0          0          0          0          0          0          0          0
 12        679          0          0          0          0          0          0          0          0          0          0          0
 13         92          0          0          0          0          0          0          0          0          0          0          0
 14        666          0          0          0          0          0          0          0          0          0          0          0
 15        106          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.551s ( 1082.882 blocks/s,  67.680 Mb/s)
  358   Sun Jun 6 14:15:05 2021 TDSunday 6 June
14.15 DAQ continues file S496/R32_117
      Hot HEC channels multiple FEE64s from c. 06.00 this morning
      ASIC check fixes

      System wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15     12     12      3      3      4      2      2      1      4      7   : 40572
aida02 :      7      8     10      5      2      3      3      2      1      4      7   : 40604
aida03 :     27     10      4      8      1      4      1      3      2      3      7   : 39740
aida04 :     18      7      4      5      1      3      2      3      2      3      7   : 39712
aida05 :     15     13      8      4      1      3      1      3      1      4      7   : 40548
aida06 :     14      8      8      6      3      2      1      3      1      4      7   : 40568
aida07 :     11      4      5      6      2      3      3      2      1      4      7   : 40540
aida08 :     15      6      5      3      4      3      3      2      1      4      7   : 40604
aida09 :     18      4      4      3      3      2      2      3      1      4      7   : 40648
aida10 :     12      6      8      6      1      1      1      3      1      4      7   : 40288
aida11 :     16      8      6      6      4      4      3      3      2      3      7   : 40352
aida12 :     10      7      4      8      3      4      3      3      2      3      7   : 40288
aida13 :     15      7      5      4      5      4      3      3      2      3      7   : 40324
aida14 :      9      4      6      7      3      4      3      3      2      3      7   : 40260
aida15 :     12      4      8      3      2      2      2      4      1      4      7   : 41136
aida16 :     11      8      6      6      3      4      3      3      2      3      7   : 40268

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - 20us FSR - attachments 3 & 4
      ADC data items - attachment 5
      FEE64 temperatures OK - attachment 6
      DSSSD bias & leakage currents OK - attachment 7
      Merger/Tape Server/Merger statistics - attachment 8
       no merger errors reported since previous restart

16.28 analysis of file S496/R32_118 - attachment 9
       max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-06-06_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-06_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-06_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-06-06_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-06_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-06-06_Statistics_aidas-gsi.png
Screenshot_2021-06-06_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-06-06_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-06_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 984.png
984.png
Attachment 8: 985.png
985.png
Attachment 9: R32_118
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   48860593 (   17844.1 Hz)
        Other data format:  213059412 (   77810.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        101 (       0.0 Hz)
                               RESUME:        101 (       0.0 Hz)
                              SYNC100:      42200 (      15.4 Hz)
                              WR48-63:      42200 (      15.4 Hz)
                           FEE64 disc:    1112345 (     406.2 Hz)
                             MBS info:  211862465 (   77373.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2738.188 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2613.952
  2                0.575                0.000
  3                7.745                0.000
  4                0.000                0.000
  5                0.000             2154.815
  6                0.000                0.000
  7                0.000             1921.162
  8                0.000              777.447
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        155          0          0          0          0          0          0          0          0          0          0          0
  1      66625         28          0          0          0          0         14         14          0          0          0          0
  2        415  106108157          0          0         26         26      18899      18899          0  106070307          0          0
  3   39298810  106947309          0          0         75         75      21328      21328    1112345  105792158          0          0
  4        171          0          0          0          0          0          0          0          0          0          0          0
  5    4255149       1684          0          0          0          0        842        842          0          0          0          0
  6        596          0          0          0          0          0          0          0          0          0          0          0
  7    5234928       2226          0          0          0          0       1113       1113          0          0          0          0
  8        493          4          0          0          0          0          2          2          0          0          0          0
  9         92          0          0          0          0          0          0          0          0          0          0          0
 10       1212          2          0          0          0          0          1          1          0          0          0          0
 11        160          0          0          0          0          0          0          0          0          0          0          0
 12        781          0          0          0          0          0          0          0          0          0          0          0
 13         89          0          0          0          0          0          0          0          0          0          0          0
 14        724          2          0          0          0          0          1          1          0          0          0          0
 15        193          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   28.738s ( 1113.497 blocks/s,  69.594 Mb/s)
  357   Sat Jun 5 11:20:15 2021 TDSaturday 5 June
12.25 DAQ continues OK - file S496/R32_48
      alpha background

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8946 : 	 2  
aida02 fault 	 0x9b36 : 	 0x9b38 : 	 2  
aida03 fault 	 0xf26b : 	 0xf26d : 	 2  
aida04 fault 	 0x57a0 : 	 0x57a2 : 	 2  
aida05 fault 	 0x4d3c : 	 0x4d3d : 	 1  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x640d : 	 1  
aida07 fault 	 0x255e : 	 0x255f : 	 1  
aida08 fault 	 0xffbd : 	 0xffbe : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     10      6      1      2      4      4      2      2      1      4      7   : 40360
aida02 :      4      3      3      3      3      4      2      2      1      4      7   : 40312
aida03 :     13      5      4      4      2      4      1      3      2      3      7   : 39580
aida04 :     16      5      3      3      2      4      1      3      2      3      7   : 39544
aida05 :      8      6      1      2      3      3      1      3      1      4      7   : 40416
aida06 :      4      3      2      1      3      3      1      3      1      4      7   : 40360
aida07 :      6      6      2      4      4      2      3      2      1      4      7   : 40424
aida08 :      6      4      1      1      4      3      3      2      1      4      7   : 40424
aida09 :     16      1      3      1      0      2      2      3      1      4      7   : 40344
aida10 :      2      5      1      4      1      3      2      4      2      3      7   : 40064
aida11 :     14      3      2      1      4      4      3      3      2      3      7   : 40080
aida12 :     11      2      2      4      4      4      3      3      2      3      7   : 40156
aida13 :     11      3      2      2      3      4      3      3      2      3      7   : 40036
aida14 :      6      5      0      4      3      4      3      3      2      3      7   : 40064
aida15 :     11      6      3      1      3      1      2      4      1      4      7   : 40940
aida16 :     12      5      0      4      4      4      3      3      2      3      7   : 40152

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      1.8.W spectra - 20us FSR - attachments 2 & 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

      analysis of file S496/R32_47 - attachment 8
       max deadtime 0.35% aida04

13.56 Lost activity monitor - see attachment 9
Attachment 1: Screenshot_2021-06-05_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-05_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-05_Spectrum_Browser_aidas-gsi.png
Attachment 4: Screenshot_2021-06-05_Statistics_aidas-gsi.png
Screenshot_2021-06-05_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-06-05_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-05_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 982.png
982.png
Attachment 7: 983.png
983.png
Attachment 8: R32_47
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   47883982 (   17391.8 Hz)
        Other data format:  214036019 (   77739.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        184 (       0.1 Hz)
                               RESUME:        184 (       0.1 Hz)
                              SYNC100:      42255 (      15.3 Hz)
                              WR48-63:      42255 (      15.3 Hz)
                           FEE64 disc:    1053753 (     382.7 Hz)
                             MBS info:  212897388 (   77325.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2753.252 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2696.425
  2                0.825                0.000
  3                9.735                0.000
  4                0.000                0.000
  5                0.000             2175.120
  6                0.000                0.000
  7                0.000             2003.563
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        187          0          0          0          0          0          0          0          0          0          0          0
  1      86597         40          0          0          0          0         20         20          0          0          0          0
  2        381  106659018          0          0         37         37      18746      18746          0  106621452          0          0
  3   38584230  107373193          0          0        147        147      21605      21605    1053753  106275936          0          0
  4        143          0          0          0          0          0          0          0          0          0          0          0
  5    4208337       1756          0          0          0          0        878        878          0          0          0          0
  6        545          0          0          0          0          0          0          0          0          0          0          0
  7    5000124       2012          0          0          0          0       1006       1006          0          0          0          0
  8        478          0          0          0          0          0          0          0          0          0          0          0
  9         96          0          0          0          0          0          0          0          0          0          0          0
 10       1149          0          0          0          0          0          0          0          0          0          0          0
 11        117          0          0          0          0          0          0          0          0          0          0          0
 12        706          0          0          0          0          0          0          0          0          0          0          0
 13        105          0          0          0          0          0          0          0          0          0          0          0
 14        636          0          0          0          0          0          0          0          0          0          0          0
 15        151          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.996s ( 1066.806 blocks/s,  66.675 Mb/s)
Attachment 9: Screenshot_2021-06-05_Lost_activity_monitor_aidas-gsi(1).png
Screenshot_2021-06-05_Lost_activity_monitor_aidas-gsi(1).png
  356   Fri Jun 4 09:47:44 2021 TDaida15 - system console output - 00.02 Friday 4 June reboot
03/23:58:55|kernel BUG at mm/slab.c:2974!
03/23:58:55|Oops: Exception in kernel mode, sig: 5 [#1]
03/23:58:55|PREEMPT Xilinx Virtex440
03/23:58:55|Modules linked in: aidamem xdriver xh_spidev_register
03/23:58:55|NIP: c009211c LR: c00920b0 CTR: c0038e70
03/23:58:55|REGS: c631fc60 TRAP: 0700   Not tainted  (2.6.31)
03/23:58:55|MSR: 00021000 <ME,CE>  CR: 22022048  XER: 00000000
03/23:58:56|TASK = c627a0c0[375] 'AidaExecV9' THREAD: c631e000
03/23:58:56|GPR00: 00000001 c631fd10 c627a0c0 c680daf0 00000020 00000001 c6940020 00000000 
03/23:58:56|GPR08: fffffff6 00010002 c680e400 c631e000 22008022 1005520c c03a0000 00000020 
03/23:58:56|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 
03/23:58:56|GPR24: 00100100 c631e000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400 
03/23:58:56|NIP [c009211c] cache_alloc_refill+0x130/0x608
03/23:58:56|LR [c00920b0] cache_alloc_refill+0xc4/0x608
03/23:58:56|Call Trace:
03/23:58:56|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
03/23:58:56|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
03/23:58:56|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8
03/23:58:56|[c631fdb0] [c0042938] __send_signal+0x78/0x260
03/23:58:56|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c
03/23:58:56|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c
03/23:58:56|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30
03/23:58:56|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240
03/23:58:56|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c
03/23:58:56|[c631feb0] [c0040bf4] update_process_times+0x30/0x70
03/23:58:57|[c631fed0] [c005a000] tick_periodic+0x34/0xe8
03/23:58:57|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120
03/23:58:57|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c
03/23:58:57|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18
03/23:58:57|Instruction dump:
03/23:58:57|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 
03/23:58:57|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 
03/23:58:57|Kernel panic - not syncing: Fatal exception in interrupt
03/23:58:57|Call Trace:
03/23:58:57|[c631fab0] [c0005de8] show_stack+0x44/0x16c (unreliable)
03/23:58:57|[c631faf0] [c00345bc] panic+0x94/0x168
03/23:58:57|[c631fb40] [c000bd44] die+0x178/0x18c
03/23:58:57|[c631fb60] [c000c000] _exception+0x164/0x1b4
03/23:58:57|[c631fc50] [c000e978] ret_from_except_full+0x0/0x4c
03/23:58:57|[c631fd10] [c00920b0] cache_alloc_refill+0xc4/0x608
03/23:58:57|[c631fd70] [c00927d8] kmem_cache_alloc+0xc4/0xcc
03/23:58:57|[c631fd90] [c0042420] __sigqueue_alloc+0x50/0xb8
03/23:58:57|[c631fdb0] [c0042938] __send_signal+0x78/0x260
03/23:58:57|[c631fde0] [c0042f78] group_send_sig_info+0x70/0x9c
03/23:58:58|[c631fe10] [c00438a8] kill_pid_info+0x48/0x8c
03/23:58:58|[c631fe30] [c0038e8c] it_real_fn+0x1c/0x30
03/23:58:58|[c631fe40] [c0050c40] hrtimer_run_queues+0x184/0x240
03/23:58:58|[c631fea0] [c0040ba8] run_local_timers+0x10/0x2c
03/23:58:58|[c631feb0] [c0040bf4] update_process_times+0x30/0x70
03/23:58:58|[c631fed0] [c005a000] tick_periodic+0x34/0xe8
03/23:58:58|[c631fee0] [c005a0d4] tick_handle_periodic+0x20/0x120
03/23:58:58|[c631ff20] [c000af70] timer_interrupt+0xa4/0x10c
03/23:58:58|[c631ff40] [c000e9c4] ret_from_except+0x0/0x18
03/23:58:58|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
04/00:01:59|
Found 0 errors checking kernel image
04/00:02:00|VHDL version number 0X03350706 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
04/00:02:00|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
04/00:02:00|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
04/00:02:00|SPIconfig done now to set up the LMK3200 registers
04/00:02:00|LMK #0 : regInit[0]=0x80000000
04/00:02:00|LMK #0 : regInit[1]=0x10070600
04/00:02:00|LMK #0 : regInit[2]=0x60601
04/00:02:00|LMK #0 : regInit[3]=0x60602
04/00:02:00|LMK #0 : regInit[4]=0x60603
04/00:02:00|LMK #0 : regInit[5]=0x70624
04/00:02:00|LMK #0 : regInit[6]=0x70605
04/00:02:00|LMK #0 : regInit[7]=0x70606
04/00:02:00|LMK #0 : regInit[8]=0x70627
04/00:02:00|LMK #0 : regInit[9]=0x10000908
04/00:02:00|LMK #0 : regInit[10]=0xA0022A09
04/00:02:00|LMK #0 : regInit[11]=0x82800B
04/00:02:01|LMK #0 : regInit[12]=0x28C800D
04/00:02:01|LMK #0 : regInit[13]=0x830020E
04/00:02:01|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
04/00:02:01|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
04/00:02:01|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
04/00:02:01|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
04/00:02:01|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
04/00:02:05|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xee 0x10 
04/00:02:11|Using Xilinx Virtex440 machine description
04/00:02:12|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
04/00:02:12|Zone PFN ranges:
04/00:02:12|  DMA      0x00000000 -> 0x00007000
04/00:02:12|  Normal   0x00007000 -> 0x00007000
04/00:02:12|Movable zone start PFN for each node
04/00:02:12|early_node_map[1] active PFN ranges
04/00:02:12|    0: 0x00000000 -> 0x00007000
04/00:02:12|MMU: Allocated 1088 bytes of context maps for 255 contexts
04/00:02:12|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
04/00:02:12|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
04/00:02:12|PID hash table entries: 512 (order: 9, 2048 bytes)
04/00:02:12|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
04/00:02:12|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
04/00:02:12|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
04/00:02:12|Kernel virtual memory layout:
04/00:02:12|  * 0xffffe000..0xfffff000  : fixmap
04/00:02:12|  * 0xfde00000..0xfe000000  : consistent mem
04/00:02:12|  * 0xfde00000..0xfde00000  : early ioremap
04/00:02:13|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
04/00:02:13|NR_IRQS:512
04/00:02:13|clocksource: timebase mult[a00000] shift[22] registered
04/00:02:13|Console: colour dummy device 80x25
04/00:02:13|Mount-cache hash table entries: 512
04/00:02:13|NET: Registered protocol family 16
04/00:02:13|PCI: Probing PCI hardware
04/00:02:13|bio: create slab <bio-0> at 0
04/00:02:13|NET: Registered protocol family 2
04/00:02:13|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
04/00:02:13|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
04/00:02:13|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
04/00:02:13|TCP: Hash tables configured (established 4096 bind 4096)
04/00:02:13|TCP reno registered
04/00:02:13|NET: Registered protocol family 1
04/00:02:13|ROMFS MTD (C) 2007 Red Hat, Inc.
04/00:02:13|msgmni has been set to 214
04/00:02:13|io scheduler noop registered
04/00:02:13|io scheduler anticipatory registered
04/00:02:13|io scheduler deadline registered
04/00:02:13|io scheduler cfq registered (default)
04/00:02:13|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
04/00:02:13|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
04/00:02:14|console [ttyS0] enabled
04/00:02:14|brd: module loaded
04/00:02:14|loop: module loaded
04/00:02:14|Device Tree Probing 'ethernet'
04/00:02:14|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
04/00:02:14|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
04/00:02:14|XLlTemac: DCR address: 0x80
04/00:02:14|XLlTemac: buffer descriptor size: 32768 (0x8000)
04/00:02:14|XLlTemac: Allocating DMA descriptors with kmalloc
04/00:02:14|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
04/00:02:14|XTemac: PHY detected at address 7.
04/00:02:14|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
04/00:02:14|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14| Intel/Sharp Extended Query Table at 0x010A
04/00:02:14|Using buffer write method
04/00:02:15|cfi_cmdset_0001: Erase suspend on write enabled
04/00:02:15|cmdlinepart partition parsing not available
04/00:02:15|RedBoot partition parsing not available
04/00:02:15|Creating 5 MTD partitions on "fc000000.flash":
04/00:02:15|0x000000000000-0x000000500000 : "golden_firmware"
04/00:02:15|0x000000500000-0x000000800000 : "golden_kernel"
04/00:02:15|0x000000800000-0x000000d00000 : "user_firmware"
04/00:02:15|0x000000d00000-0x000000fe0000 : "user_kernel"
04/00:02:15|0x000000fe0000-0x000001000000 : "env_variables"
04/00:02:15|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
04/00:02:15|SPI: XIlinx spi: bus number now 32766
04/00:02:15|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
04/00:02:15|SPI: XIlinx spi: bus number now 32765
04/00:02:15|mice: PS/2 mouse device common for all mice
04/00:02:15|Device Tree Probing 'i2c'
04/00:02:15| #0 at 0x81600000 mapped to 0xD1030000, irq=22
04/00:02:15|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
04/00:02:15|TCP cubic registered
04/00:02:15|NET: Registered protocol family 17
04/00:02:15|RPC: Registered udp transport module.
04/00:02:15|RPC: Registered tcp transport module.
04/00:02:16|eth0: XLlTemac: Options: 0x3fa
04/00:02:16|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
04/00:02:16|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
04/00:02:16|eth0: XLlTemac: speed set to 1000Mb/s
04/00:02:18|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
04/00:02:18|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
04/00:02:18|Sending DHCP requests ., OK
04/00:02:19|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.15
04/00:02:20|IP-Config: Complete:
04/00:02:20|     device=eth0, addr=192.168.11.15, mask=255.255.255.0, gw=255.255.255.255,
04/00:02:20|     host=aida15, domain=dl.ac.uk, nis-domain=nuclear.physics,
04/00:02:20|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15
04/00:02:20|Looking up port of RPC 100003/2 on 192.168.11.99
04/00:02:20|Looking up port of RPC 100005/1 on 192.168.11.99
04/00:02:20|VFS: Mounted root (nfs filesystem) on device 0:12.
04/00:02:20|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
04/00:02:20|Starting sysinit...
04/00:02:21|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
04/00:02:22|		Press 'I' to enter interactive startup.
04/00:02:22|Setting clock  (utc): Thu Jun  3 23:02:22 BST 2021 [  OK  ]
04/00:02:22|Building the cache [  OK  ]
04/00:02:22|Setting hostname aida15:  [  OK  ]
04/00:02:23|Mounting local filesystems:  [  OK  ]
04/00:02:25|Enabling /etc/fstab swaps:  [  OK  ]
04/00:02:27|Finishing sysinit...
INIT: Entering runlevel: 3
04/00:02:30|Entering non-interactive startup
04/00:02:30|FATAL: Module ipv6 not found.
04/00:02:31|Bringing up loopback interface:  [  OK  ]
04/00:02:33|FATAL: Module ipv6 not found.
04/00:02:34|Starting system logger: [  OK  ]
04/00:02:34|Starting kernel logger: [  OK  ]
04/00:02:35|Starting rpcbind: [  OK  ]
04/00:02:36|Mounting NFS filesystems:  [  OK  ]
04/00:02:37|Mounting other filesystems:  [  OK  ]
04/00:02:37|Starting xinetd: [  OK  ]
04/00:02:38|Starting midas:  Starting MIDAS Data Acquisition for aida15
04/00:02:38|xaida: device parameters: base=0x81000000 size=0x200000
04/00:02:44|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
04/00:02:44|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
04/00:02:44|xaida: driver assigned major number 254
04/00:02:44|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
04/00:02:50|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
04/00:02:50|AIDAMEM: aidamem: driver assigned major number 253
04/00:02:50|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
04/00:02:56|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
04/00:02:57|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
04/00:02:57|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
04/00:02:57|Computer Name = aida15; Temp Directory = /tmp/tcl361
04/00:03:01|package limit is not available: can't find package limit
04/00:03:04|Running with default file descriptor limit
04/00:03:04|package setuid is not available: can't find package setuid
04/00:03:05|Running as user 0 group 0
04/00:03:05|
04/00:03:06|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
04/00:03:06|
04/00:03:06|Built without pthreads
04/00:03:06|
04/00:03:06|Creating NetVars
04/00:03:06|Output buffer length = 65xaida: open:
04/00:03:06|504; format option = 4AIDAMEM: aidamem_open:
04/00:03:06|; transfer option = 3
04/00:03:06|EB transfer option = 3
04/00:03:06|NetVars created and initialised
04/00:03:06|Statistics thread starting
04/00:03:06|Statistics thread created
04/00:03:06|Stat/Rate creation thread starting
04/00:03:06|Data Acquisition task has PID 375
04/00:03:06|Stat/Rate creation thread created
04/00:03:06|Hit/Rate creation thread starting
04/00:03:06|Hit/Rate creation thread created
04/00:03:07|AIDA Heartbeat thread starting
04/00:03:07|Heartbeat thread created
04/00:03:07|Installing signal handlers
04/00:03:07|Done
04/00:03:07|ModuleNum = 0
04/00:03:07|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 03350706 
04/00:03:07|Spectra table initialised
04/00:03:07|AIDA Data Acquisition now all ready to start
04/00:03:07|SIGBUS, SIGSEGV and SIGPIPE traps setup
04/00:03:07|/debug user "debug" password " bjkhcfyuqis"
04/00:03:14|httpd started on port 8015
04/00:03:14|
04/00:03:14|Cannot use /MIDAS/config/TclHttpd/aida15@8015/startup.tcl
04/00:03:14|Custom startup from /MIDAS/config/TclHttpd/aida15/startup.tcl
04/00:03:14|XAIDA Access package 1.0
04/00:03:16|/XAIDAAccessServer
04/00:03:16|XAD9252 Access package 1.0
04/00:03:16|/XAD9252AccessServer
04/00:03:16|/DataBaseAccessServer
04/00:03:16|[  OK  ]
04/00:03:16|/NetVarService
04/00:03:17|/SigTaskService
04/00:03:17|Loaded MemSasAccess
04/00:03:17|/SpectrumService
04/00:03:17|loading tcl/AIDARunControl.tcl for namespace ::
04/00:03:17|
04/00:03:17|DENX ELDK version 4.2 build 2008-04-01
04/00:03:17|Linux 2.6.31 on a ppc
04/00:03:17|
04/00:03:17|aida15 login: /DataAcquisitionControlServer
04/00:03:17|DefineMessage unknown
04/00:03:17|Run Control Server Implementation for AIDA
04/00:03:17|RunControlServer loaded
04/00:03:17|loading Html/RunControl/implementation.tcl
04/00:03:18|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
04/00:03:18|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
04/00:03:18|Created UI registers
04/00:03:18|RunControl loaded
04/00:03:19|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
04/00:03:19|AIDA RunControl loaded
04/00:03:19|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
04/00:03:19|do_GetState returned z=0 and 8
04/09:31:05|do_GetState returned z=0 and 8
04/09:31:17|do_GetState returned z=0 and 8
04/09:31:30|do_GetState returned z=0 and 8
04/09:31:43|do_GetState returned z=0 and 8
04/09:31:55|do_GetState returned z=0 and 8
04/09:32:07|do_GetState returned z=0 and 8
04/09:32:19|do_GetState returned z=0 and 8
04/09:32:32|do_GetState returned z=0 and 8
04/09:32:45|do_GetState returned z=0 and 8
04/09:32:57|do_GetState returned z=0 and 8
04/09:33:10|do_GetState returned z=0 and 8
04/09:33:22|do_GetState returned z=0 and 8
04/09:33:35|do_GetState returned z=0 and 8
04/09:33:47|do_GetState returned z=0 and 8
04/09:34:00|do_GetState returned z=0 and 8
04/09:34:13|do_GetState returned z=0 and 8
04/09:34:25|do_GetState returned z=0 and 8
04/09:34:38|do_GetState returned z=0 and 8
04/09:34:50|do_GetState returned z=0 and 8
04/09:35:03|do_GetState returned z=0 and 8
04/09:35:16|do_GetState returned z=0 and 8
04/09:35:28|do_GetState returned z=0 and 8
04/09:35:41|do_GetState returned z=0 and 8
04/09:35:53|do_GetState returned z=0 and 8
04/09:36:06|do_GetState returned z=0 and 8
04/09:36:18|do_GetState returned z=0 and 8
04/09:36:31|do_GetState returned z=0 and 8
04/09:36:43|do_GetState returned z=0 and 8
04/09:36:56|do_GetState returned z=0 and 8
04/09:37:09|do_GetState returned z=0 and 8
04/09:37:21|do_GetState returned z=0 and 8
04/09:37:34|do_GetState returned z=0 and 8
04/09:37:46|do_GetState returned z=0 and 8
04/09:37:59|do_GetState returned z=0 and 8
04/09:38:12|do_GetState returned z=0 and 8
04/09:38:24|do_GetState returned z=0 and 8
04/09:38:37|do_GetState returned z=0 and 8
04/09:38:49|do_GetState returned z=0 and 8
04/09:39:02|do_GetState returned z=0 and 8
04/09:39:15|do_GetState returned z=0 and 8
04/09:39:27|do_GetState returned z=0 and 8
04/09:39:40|do_GetState returned z=0 and 8
04/09:39:52|do_GetState returned z=0 and 8
04/09:40:05|do_GetState returned z=0 and 8
04/09:40:18|do_GetState returned z=0 and 8
04/09:40:30|do_GetState returned z=0 and 8
04/09:40:43|do_GetState returned z=0 and 8
04/09:40:55|do_GetState returned z=0 and 8
04/09:41:08|do_GetState returned z=0 and 8
04/09:41:21|do_GetState returned z=0 and 8
04/09:41:33|do_GetState returned z=0 and 8
04/09:41:46|do_GetState returned z=0 and 8
04/09:41:58|do_GetState returned z=0 and 8
04/09:42:11|do_GetState returned z=0 and 8
04/09:42:23|do_GetState returned z=0 and 8
04/09:42:36|do_GetState returned z=0 and 8
04/09:42:48|do_GetState returned z=0 and 8
04/09:43:01|do_GetState returned z=0 and 8
04/09:43:14|do_GetState returned z=0 and 8
04/09:43:26|do_GetState returned z=0 and 8
04/09:43:39|do_GetState returned z=0 and 8
04/09:43:51|do_GetState returned z=0 and 8
04/09:44:04|do_GetState returned z=0 and 8
04/09:44:17|do_GetState returned z=0 and 8
04/09:44:29|do_GetState returned z=0 and 8
04/09:44:42|do_GetState returned z=0 and 8
04/09:44:54|do_GetState returned z=0 and 8
04/09:45:07|do_GetState returned z=0 and 8
04/09:45:19|do_GetState returned z=0 and 8
04/09:45:32|do_GetState returned z=0 and 8
04/09:45:44|do_GetState returned z=0 and 8
04/09:45:57|do_GetState returned z=0 and 8
04/09:46:09|do_GetState returned z=0 and 8
04/09:46:22|do_GetState returned z=0 and 8
04/09:46:34|do_GetState returned z=0 and 8
04/09:46:47|do_GetState returned z=0 and 8
04/09:47:00|do_GetState returned z=0 and 8
04/09:47:12|do_GetState returned z=0 and 8
04/09:47:25|do_GetState returned z=0 and 8
04/09:47:37|do_GetState returned z=0 and 8
04/09:47:50|do_GetState returned z=0 and 8
04/09:48:03|do_GetState returned z=0 and 8
04/09:48:15|do_GetState returned z=0 and 8
04/09:48:28|do_GetState returned z=0 and 8
04/09:48:40|do_GetState returned z=0 and 8
04/09:48:53|do_GetState returned z=0 and 8
04/09:49:05|do_GetState returned z=0 and 8
04/09:49:18|do_GetState returned z=0 and 8
04/09:49:31|do_GetState returned z=0 and 8
04/09:49:43|do_GetState returned z=0 and 8
04/09:49:56|do_GetState returned z=0 and 8
04/09:50:08|do_GetState returned z=0 and 8
04/09:50:21|do_GetState returned z=0 and 8
04/09:50:33|do_GetState returned z=0 and 8
04/09:50:46|do_GetState returned z=0 and 8
04/09:50:58|do_GetState returned z=0 and 8
04/09:51:11|do_GetState returned z=0 and 8
04/09:51:23|do_GetState returned z=0 and 8
04/09:51:36|do_GetState returned z=0 and 8
04/09:51:49|do_GetState returned z=0 and 8
04/09:52:01|do_GetState returned z=0 and 8
04/09:52:14|do_GetState returned z=0 and 8
04/09:52:26|do_GetState returned z=0 and 8
04/09:52:39|do_GetState returned z=0 and 8
04/09:52:51|do_GetState returned z=0 and 8
04/09:53:04|do_GetState returned z=0 and 8
04/09:53:16|do_GetState returned z=0 and 8
04/09:53:29|do_GetState returned z=0 and 8
04/09:53:42|do_GetState returned z=0 and 8
04/09:53:54|do_GetState returned z=0 and 8
04/09:54:07|do_GetState returned z=0 and 8
04/09:54:19|do_GetState returned z=0 and 8
04/09:54:32|do_GetState returned z=0 and 8
04/09:54:44|do_GetState returned z=0 and 8
04/09:54:57|do_GetState returned z=0 and 8
04/09:55:09|do_GetState returned z=0 and 8
04/09:55:22|do_GetState returned z=0 and 8
04/09:55:34|do_GetState returned z=0 and 8
04/09:55:47|do_GetState returned z=0 and 8
04/09:56:00|do_GetState returned z=0 and 8
04/09:56:12|do_GetState returned z=0 and 8
04/09:56:25|do_GetState returned z=0 and 8
04/09:56:37|do_GetState returned z=0 and 8
04/09:56:50|do_GetState returned z=0 and 8
04/09:57:02|do_GetState returned z=0 and 8
04/09:57:15|do_GetState returned z=0 and 8
04/09:57:27|do_GetState returned z=0 and 8
04/09:57:40|do_GetState returned z=0 and 8
04/09:57:53|do_GetState returned z=0 and 8
04/09:58:05|do_GetState returned z=0 and 8
04/09:58:18|do_GetState returned z=0 and 8
04/09:58:30|do_GetState returned z=0 and 8
04/09:58:43|do_GetState returned z=0 and 8
04/09:58:56|do_GetState returned z=0 and 8
04/09:59:08|do_GetState returned z=0 and 8
04/09:59:21|do_GetState returned z=0 and 8
04/09:59:33|do_GetState returned z=0 and 8
04/09:59:46|do_GetState returned z=0 and 8
04/09:59:58|do_GetState returned z=0 and 8
04/10:00:11|do_GetState returned z=0 and 8
04/10:00:23|do_GetState returned z=0 and 8
04/10:00:36|do_GetState returned z=0 and 8
04/10:00:48|do_GetState returned z=0 and 8
04/10:01:01|do_GetState returned z=0 and 8
04/10:01:13|do_GetState returned z=0 and 8
04/10:01:26|do_GetState returned z=0 and 8
04/10:01:38|do_GetState returned z=0 and 8
04/10:01:51|do_GetState returned z=0 and 8
04/10:02:04|do_GetState returned z=0 and 8
04/10:02:16|do_GetState returned z=0 and 8
04/10:02:29|do_GetState returned z=0 and 8
04/10:02:41|do_GetState returned z=0 and 8
04/10:02:54|do_GetState returned z=0 and 8
04/10:03:06|do_GetState returned z=0 and 8
04/10:03:19|do_GetState returned z=0 and 8
04/10:03:31|do_GetState returned z=0 and 8
04/10:03:44|do_GetState returned z=0 and 8
04/10:03:57|do_GetState returned z=0 and 8
04/10:04:09|do_GetState returned z=0 and 8
04/10:04:22|do_GetState returned z=0 and 8
04/10:04:34|do_GetState returned z=0 and 8
04/10:04:47|do_GetState returned z=0 and 8
04/10:04:59|do_GetState returned z=0 and 8
04/10:05:12|do_GetState returned z=0 and 8
04/10:05:25|do_GetState returned z=0 and 8
04/10:05:37|do_GetState returned z=0 and 8
04/10:05:50|do_GetState returned z=0 and 8
04/10:06:02|do_GetState returned z=0 and 8
04/10:06:15|do_GetState returned z=0 and 8
04/10:06:27|do_GetState returned z=0 and 8
04/10:06:40|do_GetState returned z=0 and 8
04/10:06:52|do_GetState returned z=0 and 8
04/10:07:05|do_GetState returned z=0 and 8
04/10:07:18|do_GetState returned z=0 and 8
04/10:07:30|do_GetState returned z=0 and 8
  355   Fri Jun 4 08:40:52 2021 TDFriday 4 June
09.20 DAQ found stopped
      files in /TapeData/S496 indicate increased data rate yesterday evening from c. 18.00 until data stopped at c. midnight
      aida15 rebooted at 00.02

      regained control of DAQ and issued DAQ STOP by restarting NewMerger *twice*

11.00 DAQ reset and setup
      All system wide checks OK *except*

		 Base 		Current 	Difference
aida05 fault 	 0x4d3c : 	 0x4d3c : 	 0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      4      2      8      5      4      1      2      2      4      7   : 41352
aida02 :      0      3      3      6      6      4      1      2      2      4      7   : 41352
aida03 :      2      4      2     12      1      3      2      2      2      4      7   : 41352
aida04 :      2      2      1      7      4      3      2      2      2      4      7   : 41352
aida05 :      2      3      1      5      3      2      3      2      2      4      7   : 41360
aida06 :      2      4      2      6      4      3      2      2      2      4      7   : 41352
aida07 :      1      5      2      9      4      2      4      3      1      4      7   : 41324
aida08 :      0      3      1      7      4      3      4      3      1      4      7   : 41352
aida09 :      3      3      0      8      2      2      3      2      2      4      7   : 41380
aida10 :      1      2      2     11      3      2      3      3      3      3      7   : 41044
aida11 :      2      3      1      6      6      4      2      3      3      3      7   : 41072
aida12 :      4      2      1      8      3      3      3      3      1      4      7   : 41072
aida13 :      2      4      1      8      4      4      2      3      3      3      7   : 41016
aida14 :      1      4      1      9      4      4      2      3      3      3      7   : 41044
aida15 :      0      5      1      2      2      1      2      4      3      4      7   : 42872
aida16 :      1      3      2      9      4      4      4      4      2      3      7   : 41052

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Jun 04 09:30:42 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

11.03 DAQ resumes file S496/R32
      alpha background

      slow comparator 0x64
      BNC PB-5 pulser OFF

11.09 1.8.W spectra - 20us FSR - attachments 1 & 2
      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

13.07 analysis of file S496/R32_16 - attachment 8
       max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi(1).png
Attachment 2: Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-04_Spectrum_Browser_aidas-gsi.png
Attachment 3: Screenshot_2021-06-04_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-04_AIDA_Alerting_-_Grafana.png
Attachment 4: Screenshot_2021-06-04_Statistics_aidas-gsi.png
Screenshot_2021-06-04_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-06-04_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-04_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 980.png
980.png
Attachment 7: 981.png
981.png
Attachment 8: R32_16
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   49166189 (   17465.0 Hz)
        Other data format:  212753815 (   75575.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        137 (       0.0 Hz)
                               RESUME:        137 (       0.0 Hz)
                              SYNC100:      42486 (      15.1 Hz)
                              WR48-63:      42486 (      15.1 Hz)
                           FEE64 disc:    1037793 (     368.6 Hz)
                             MBS info:  211630776 (   75176.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2815.130 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2759.088
  2                0.741                0.000
  3                4.946                0.000
  4                0.000                0.000
  5                0.000             2173.993
  6                0.000                0.000
  7                0.000             2104.432
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        184          0          0          0          0          0          0          0          0          0          0          0
  1     115458         62          0          0          0          0         31         31          0          0          0          0
  2        444  105932150          0          0         36         36      18754      18754          0  105894570          0          0
  3   39683483  106817705          0          0        101        101      21752      21752    1037793  105736206          0          0
  4        154          0          0          0          0          0          0          0          0          0          0          0
  5    4413391       1852          0          0          0          0        926        926          0          0          0          0
  6        586          0          0          0          0          0          0          0          0          0          0          0
  7    4948958       2044          0          0          0          0       1022       1022          0          0          0          0
  8        489          0          0          0          0          0          0          0          0          0          0          0
  9         77          0          0          0          0          0          0          0          0          0          0          0
 10       1172          0          0          0          0          0          0          0          0          0          0          0
 11        163          2          0          0          0          0          1          1          0          0          0          0
 12        754          0          0          0          0          0          0          0          0          0          0          0
 13         94          0          0          0          0          0          0          0          0          0          0          0
 14        605          0          0          0          0          0          0          0          0          0          0          0
 15        177          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   29.027s ( 1102.409 blocks/s,  68.901 Mb/s)
  354   Thu Jun 3 10:16:39 2021 TDThursday 3 June
11.19 Hot HEC channels multiple FEE64s
      ASIC check fixes

      System wide checks OK * except*

		 Base 		Current 	Difference
aida01 fault 	 0x8940 : 	 0x8941 : 	 1  
aida02 fault 	 0x9b32 : 	 0x9b33 : 	 1  
aida03 fault 	 0xf266 : 	 0xf267 : 	 1  
aida04 fault 	 0x579b : 	 0x579c : 	 1  
White Rabbit error counter test result: Passed 12, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      4      2      2      2      2      2      3      2      4      7   : 41516
aida02 :     10      7      2      0      1      2      2      3      2      4      7   : 41408
aida03 :     10      8      2      2      2      4      1      4      2      3      7   : 40008
aida04 :      5      4      4      2      2      3      1      4      2      3      7   : 39860
aida05 :     12      3      2      2      1      2      2      3      2      4      7   : 41448
aida06 :     15      9      2      2      3      2      1      3      2      4      7   : 41380
aida07 :     12      8      1      2      2      3      3      2      2      4      7   : 41408
aida08 :      9      6      3      1      2      3      3      2      2      4      7   : 41380
aida09 :     15      6      1      1      2      2      2      3      2      4      7   : 41500
aida10 :     15      6      3      2      1      3      3      2      1      4      7   : 40348
aida11 :      8      2      1      1      1      3      1      3      2      4      7   : 41248
aida12 :     18      3      2      1      3      3      2      2      2      4      7   : 41184
aida13 :      7      4      1      4      3      4      3      3      1      4      7   : 41100
aida14 :     18      6      1      4      3      4      2      2      1      4      7   : 40392
aida15 :      7      7      1      3      1      2      1      3      2      4      7   : 41220
aida16 :      4      4      2      3      3      3      2      4      1      4      7   : 41200

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Jun 02 16:27:30 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      1.8.W spectra - 20us FSR - attachments 2 & 3
      ADC data items - attachment 4
      FEE64 temperatures OK - attachment 5
      DSSSD bias & leakage currents OK - attachment 6
      Merger/Tape Server/Merger statistics - attachment 7
       no merger errors reported since previous restart

11.43 Fixed merger issue which stopped output from merger - see https://elog.ph.ed.ac.uk/DESPEC/36

13.09 analysis of file S496/R31_16 - attachment 8
       max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-06-03_AIDA_Alerting_-_Grafana.png
Screenshot_2021-06-03_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-06-03_Spectrum_Browser_aidas-gsi.png
Attachment 4: Screenshot_2021-06-03_Statistics_aidas-gsi.png
Screenshot_2021-06-03_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-06-03_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-06-03_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 970.png
970.png
Attachment 7: 971.png
971.png
Attachment 8: R31_16
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   50135749 (   17884.4 Hz)
        Other data format:  211784253 (   75547.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        199 (       0.1 Hz)
                               RESUME:        199 (       0.1 Hz)
                              SYNC100:      42443 (      15.1 Hz)
                              WR48-63:      42443 (      15.1 Hz)
                           FEE64 disc:    1081490 (     385.8 Hz)
                             MBS info:  210617479 (   75131.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          4 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2803.320 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2522.967
  2                0.535                0.000
  3                8.636                0.000
  4                0.000                0.000
  5                0.000             2119.634
  6                0.000                0.000
  7                0.000             2067.036
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        196          2          0          0          0          0          0          0          2          0          0          2
  1     193203         74          0          0          0          0         37         37          0          0          0          0
  2        471  105497692          0          0         26         26      18605      18605          0  105460430          0          0
  3   40159698  106282506          0          0        173        173      21812      21812    1081487  105157049          0          0
  4        128          0          0          0          0          0          0          0          0          0          0          0
  5    4517663       1914          0          0          0          0        957        957          0          0          0          0
  6        641          2          0          0          0          0          1          1          0          0          0          0
  7    5260254       2060          0          0          0          0       1030       1030          0          0          0          0
  8        463          0          0          0          0          0          0          0          0          0          0          0
  9        104          0          0          0          0          0          0          0          0          0          0          0
 10       1026          2          0          0          0          0          1          1          0          0          0          0
 11        141          1          0          0          0          0          0          0          1          0          0          1
 12        786          0          0          0          0          0          0          0          0          0          0          0
 13        109          0          0          0          0          0          0          0          0          0          0          0
 14        687          0          0          0          0          0          0          0          0          0          0          0
 15        179          0          0          0          0          0          0          0          0          0          0          1
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   28.445s ( 1124.966 blocks/s,  70.310 Mb/s)
  353   Wed Jun 2 14:53:05 2021 OH, TDMerger/Tapeserver issues after reboot of aida-3
On 01/06/21 aida-3 was rebooted as the graphical user interface had frozen.
We were also unable to connected to MIDAS via ssh port forwarding.

Since the restart we have been unable to write to file.

The tapeserver will allocate/mount/open a file but nothing will be written to it.
The merger will be receiving data but never shows as having any "Current links with data".

/MIDAS currently points to lrwxrwxrwx. 1 root root 45 Jan 23  2019 /MIDAS -> /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119

The current disk structure is:
[npg@aidas-gsi npg]$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sr0     11:0    1  1024M  0 rom  
sdb      8:16   0   7.3T  0 disk 
&#9500;&#9472;sdb1   8:17   0   200M  0 part 
&#9500;&#9472;sdb2   8:18   0   500M  0 part 
&#9492;&#9472;sdb3   8:19   0   7.3T  0 part 
  &#9500;&#9472;vg_aidas2-lv_root (dm-2)
       253:2    0    50G  0 lvm  
  &#9500;&#9472;vg_aidas2-lv_home (dm-3)
       253:3    0   7.2T  0 lvm  /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
  &#9492;&#9472;vg_aidas2-lv_swap (dm-4)
       253:4    0   7.8G  0 lvm  
sdc      8:32   0   7.3T  0 disk 
&#9492;&#9472;sdc1   8:33   0   7.3T  0 part /media/ThirdDrive
sdd      8:48   0   7.3T  0 disk 
&#9492;&#9472;sdd1   8:49   0   7.3T  0 part /media/SecondDrive
sda      8:0    0 465.8G  0 disk 
&#9500;&#9472;sda1   8:1    0   200M  0 part /boot/efi
&#9500;&#9472;sda2   8:2    0   500M  0 part /boot
&#9492;&#9472;sda3   8:3    0 465.1G  0 part 
  &#9500;&#9472;vg_aidasgsi-lv_root (dm-0)
       253:0    0    50G  0 lvm  /
  &#9500;&#9472;vg_aidasgsi-lv_swap (dm-1)
       253:1    0   7.8G  0 lvm  [SWAP]
  &#9492;&#9472;vg_aidasgsi-lv_home (dm-5)
       253:5    0 407.3G  0 lvm  /home


Terminal outputs for the HTTPd, Merger and TapeServer terminals - attachments 1-3

TapeServer config file - attachment 4
Merger options - attachment 5

Note in the merger window the top bar is different to what we have seen previously - Attachment 6 and https://elog.ph.ed.ac.uk/DESPEC/210418_124830/51.png
Attachment 1: 210602_MergerTape_HTTPd_Startup
tidy up
tclsh8.5_copy2: no process killed

System identified is CPU x86_64; Platform is unix; OS is Linux and Version is 2.6.32-696.el6.x86_64
Environment selected is CPU x64_64; Platform unix; OS Linux64 and Operating System Linux64
MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/Linux64
PATH = /MIDAS/bin_Linux64:/MIDAS/TclHttpd/Linux64:/MIDAS/Linux/bin64:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/npg/bin
Computer Name = aidas-gsi; Temp Directory = /tmp/tcl25624
package limit is not available: can't find package limit
Running with default file descriptor limit
package setuid is not available: can't find package setuid
Could not change to user 50 group 50: not owner
/debug user "debug" password "z5y30nlwy2e4"
httpd started on port 8115

Custom startup from /MIDAS/config/TclHttpd/aidas-gsi@8115/startup.tcl
/DataBaseAccessServer
/NetVarService
/SigTaskService
Loaded MemSasAccess
/SpectrumService
/TapeServer
loading tcl/NewMergerControl.tcl for namespace ::
/DataAcquisitionControlServer
DefineMessage unknown
Run Control Server Implementation for MERGE
RunControlServer loaded
loading Html/RunControl/implementation.tcl
/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
RunControl loaded
loading Html/NewMerger/RunControl/implementation.tcl for namespace ::
ReadRegister failed: Name=NetVar.MERGE.ID; Code= 0x10004; Info= Register name does not exist
Created UI registers
NewMerge Control loaded
Completed custom startup from /MIDAS/TclHttpd/Html/NewMerger/RunControl/stats.defn.tcl
Shared memory area located at 0x7f34d3028000
Tape Server comms table located at 0x7f34d3028000
MERGE command  30
setup MERGER
Stop Merge
Halt (1)
Action has completed: MergeCommand 2 Halt
Setup Merge
Setup (1)
Action has completed: MergeCommand 1 Setup
MERGE command  32
go MERGER
Go Merge
Go (1)
Action has completed: MergeCommand 3 Go
link 0 GO (1)
Action has completed: MergeCommand 22 {link 0 GO} 0 2
link 1 GO (1)
Action has completed: MergeCommand 22 {link 1 GO} 1 2
link 2 GO (1)
Action has completed: MergeCommand 22 {link 2 GO} 2 2
link 3 GO (1)
Action has completed: MergeCommand 22 {link 3 GO} 3 2
link 4 GO (1)
Action has completed: MergeCommand 22 {link 4 GO} 4 2
link 5 GO (1)
Action has completed: MergeCommand 22 {link 5 GO} 5 2
link 6 GO (1)
Action has completed: MergeCommand 22 {link 6 GO} 6 2
link 7 GO (1)
Action has completed: MergeCommand 22 {link 7 GO} 7 2
link 8 GO (1)
Action has completed: MergeCommand 22 {link 8 GO} 8 2
link 9 GO (1)
Action has completed: MergeCommand 22 {link 9 GO} 9 2
link 10 GO (1)
Action has completed: MergeCommand 22 {link 10 GO} 10 2
link 11 GO (1)
Action has completed: MergeCommand 22 {link 11 GO} 11 2
link 12 GO (1)
Action has completed: MergeCommand 22 {link 12 GO} 12 2
link 13 GO (1)
Action has completed: MergeCommand 22 {link 13 GO} 13 2
link 14 GO (1)
Action has completed: MergeCommand 22 {link 14 GO} 14 2
link 15 GO (1)
Action has completed: MergeCommand 22 {link 15 GO} 15 2
Resume MERGER

Attachment 2: 210602_NewMerger_Startup
Tidy up
MERGE: no process killed
merge64.AD: no process killed
Starting New Merger

[1] 25659
MERGE Master (25659): Message logger not contacted.
MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
Creating NetVars
Creating NetVars complete
MERGE Master (25659): MIDAS MERGE Server 64 bit Build May 11 2021  20:32:01
MERGE Master (25659): Configuration: TCP port base=11001 ; SHM key=11000; ids=16; links=16; Hardware=0
MERGE Master (25659): File mapped object /SHM_11000 of size 536628 created
MERGE Master (25659): Memory mapping 536628 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0a99b8000.
MERGE Master (25659): File mapped object /SHM_11001 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0a39b6000.
MERGE Master (25659): File mapped object /SHM_11002 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc09d9b4000.
MERGE Master (25659): File mapped object /SHM_11003 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0979b2000.
MERGE Master (25659): File mapped object /SHM_11004 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0919b0000.
MERGE Master (25659): File mapped object /SHM_11005 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc08b9ae000.
MERGE Master (25659): File mapped object /SHM_11006 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0859ac000.
MERGE Master (25659): File mapped object /SHM_11007 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc07f9aa000.
MERGE Master (25659): File mapped object /SHM_11008 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0799a8000.
MERGE Master (25659): File mapped object /SHM_11009 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0739a6000.
MERGE Master (25659): File mapped object /SHM_11010 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc06d9a4000.
MERGE Master (25659): File mapped object /SHM_11011 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0679a2000.
MERGE Master (25659): File mapped object /SHM_11012 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc0619a0000.
MERGE Master (25659): File mapped object /SHM_11013 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc05b99e000.
MERGE Master (25659): File mapped object /SHM_11014 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc05599c000.
MERGE Master (25659): File mapped object /SHM_11015 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc04f99a000.
MERGE Master (25659): File mapped object /SHM_11016 of size 100669480 created
MERGE Master (25659): Memory mapping 100669480 bytes
MERGE Master (25659): Shared memory segment located at address 0x7fc049998000.
MERGE Master (25659): Shared memory segment located at address 0x7fc049998000.
MERGE Master (25659): Starting stats/rates task
MERGE Master (25659): Stats/Rates task has pid 25661
MERGE Master (25659): Starting 16 link tasks
MERGE Master (25659): Link task has pid 25662
MERGE Master (25659): Link task has pid 25663
MERGE Statistics/Rates (25661): Message logger not contacted.
MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
Creating NetVars
Creating NetVars complete
MERGE Master (25659): Link task has pid 25664
MERGE Data Link (25662): Message logger not contacted.
MERGE Data Link (25662): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25662): Configuration: index = 0, SHM key=11000, TCP port base = 11001, module = -1, hardware = 0
MERGE Data Link (25662): File mapped object /SHM_11000 accessed
MERGE Data Link (25662): Memory mapping 536628 bytes
MERGE Data Link (25662): Shared memory segment located at address 0x7f198ee7a000.
MERGE Data Link (25662): Allocated 71720 bytes for local input buffer
MERGE Data Link (25662): File mapped object /SHM_11001 accessed
MERGE Data Link (25662): Memory mapping 100669480 bytes
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Master (25659): Link task has pid 25665
MERGE Data Link (25663): Message logger not contacted.
MERGE Data Link (25663): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25663): Configuration: index = 1, SHM key=11000, TCP port base = 11002, module = 0, hardware = 0
MERGE Data Link (25663): File mapped object /SHM_11000 accessed
MERGE Data Link (25663): Memory mapping 536628 bytes
MERGE Data Link (25663): Shared memory segment located at address 0x7f3bee603000.
MERGE Data Link (25663): Allocated 71720 bytes for local input buffer
MERGE Data Link (25663): File mapped object /SHM_11002 accessed
MERGE Data Link (25663): Memory mapping 100669480 bytes
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Master (25659): Link task has pid 25666
MERGE Data Link (25664): Message logger not contacted.
MERGE Data Link (25665): Message logger not contacted.
MERGE Data Link (25664): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
MERGE Data Link (25665): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars
Creating NetVars complete
MERGE Data Link (25665): Configuration: index = 3, SHM key=11000, TCP port base = 11004, module = 2, hardware = 0
MERGE Data Link (25665): File mapped object /SHM_11000 accessed
MERGE Data Link (25665): Memory mapping 536628 bytes
MERGE Data Link (25665): Shared memory segment located at address 0x7f7418ae6000.
MERGE Data Link (25665): Allocated 71720 bytes for local input buffer
MERGE Data Link (25665): File mapped object /SHM_11004 accessed
MERGE Data Link (25665): Memory mapping 100669480 bytes
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
Creating NetVars complete
MERGE Data Link (25664): Configuration: index = 2, SHM key=11000, TCP port base = 11003, module = 1, hardware = 0
MERGE Data Link (25664): File mapped object /SHM_11000 accessed
MERGE Data Link (25664): Memory mapping 536628 bytes
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da72c8000.
MERGE Data Link (25664): Allocated 71720 bytes for local input buffer
MERGE Data Link (25664): File mapped object /SHM_11003 accessed
MERGE Data Link (25664): Memory mapping 100669480 bytes
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Master (25659): Link task has pid 25667
MERGE Master (25659): Link task has pid 25668
MERGE Data Link (25666): Message logger not contacted.
MERGE Data Link (25666): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25666): Configuration: index = 4, SHM key=11000, TCP port base = 11005, module = 3, hardware = 0
MERGE Data Link (25666): File mapped object /SHM_11000 accessed
MERGE Data Link (25666): Memory mapping 536628 bytes
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7f4ec2000.
MERGE Data Link (25666): Allocated 71720 bytes for local input buffer
MERGE Data Link (25666): File mapped object /SHM_11005 accessed
MERGE Data Link (25666): Memory mapping 100669480 bytes
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Master (25659): Link task has pid 25669
MERGE Data Link (25667): Message logger not contacted.
MERGE Data Link (25667): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25667): Configuration: index = 5, SHM key=11000, TCP port base = 11006, module = 4, hardware = 0
MERGE Data Link (25667): File mapped object /SHM_11000 accessed
MERGE Data Link (25667): Memory mapping 536628 bytes
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b878c9000.
MERGE Data Link (25667): Allocated 71720 bytes for local input buffer
MERGE Data Link (25667): File mapped object /SHM_11006 accessed
MERGE Data Link (25667): Memory mapping 100669480 bytes
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Master (25659): Link task has pid 25670
MERGE Data Link (25668): Message logger not contacted.
MERGE Data Link (25668): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25671
Creating NetVars complete
MERGE Data Link (25668): Configuration: index = 6, SHM key=11000, TCP port base = 11007, module = 5, hardware = 0
MERGE Data Link (25668): File mapped object /SHM_11000 accessed
MERGE Data Link (25668): Memory mapping 536628 bytes
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c7c9a000.
MERGE Data Link (25668): Allocated 71720 bytes for local input buffer
MERGE Data Link (25668): File mapped object /SHM_11007 accessed
MERGE Data Link (25668): Memory mapping 100669480 bytes
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25669): Message logger not contacted.
MERGE Data Link (25669): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25672
Creating NetVars complete
MERGE Data Link (25669): Configuration: index = 7, SHM key=11000, TCP port base = 11008, module = 6, hardware = 0
MERGE Data Link (25669): File mapped object /SHM_11000 accessed
MERGE Data Link (25669): Memory mapping 536628 bytes
MERGE Data Link (25669): Shared memory segment located at address 0x7f87e7704000.
MERGE Data Link (25669): Allocated 71720 bytes for local input buffer
MERGE Data Link (25669): File mapped object /SHM_11008 accessed
MERGE Data Link (25669): Memory mapping 100669480 bytes
MERGE Data Link (25669): Shared memory segment located at address 0x7f87e1702000.
MERGE Data Link (25671): Message logger not contacted.
MERGE Data Link (25671): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25671): Configuration: index = 9, SHM key=11000, TCP port base = 11010, module = 8, hardware = 0
MERGE Data Link (25671): File mapped object /SHM_11000 accessed
MERGE Data Link (25671): Memory mapping 536628 bytes
MERGE Data Link (25671): Shared memory segment located at address 0x7f38b4f59000.
MERGE Data Link (25671): Allocated 71720 bytes for local input buffer
MERGE Data Link (25671): File mapped object /SHM_11010 accessed
MERGE Data Link (25671): Memory mapping 100669480 bytes
MERGE Data Link (25671): Shared memory segment located at address 0x7f38aef57000.
MERGE Data Link (25670): Message logger not contacted.
MERGE Data Link (25670): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Master (25659): Link task has pid 25673
Creating NetVars complete
MERGE Data Link (25670): Configuration: index = 8, SHM key=11000, TCP port base = 11009, module = 7, hardware = 0
MERGE Data Link (25670): File mapped object /SHM_11000 accessed
MERGE Data Link (25670): Memory mapping 536628 bytes
MERGE Data Link (25672): Message logger not contacted.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff7744ce000.
MERGE Data Link (25672): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
MERGE Data Link (25670): Allocated 71720 bytes for local input buffer
MERGE Data Link (25670): File mapped object /SHM_11009 accessed
MERGE Data Link (25670): Memory mapping 100669480 bytes
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
Creating NetVars complete
MERGE Data Link (25672): Configuration: index = 10, SHM key=11000, TCP port base = 11011, module = 9, hardware = 0
MERGE Data Link (25672): File mapped object /SHM_11000 accessed
MERGE Data Link (25672): Memory mapping 536628 bytes
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bb287e000.
MERGE Data Link (25672): Allocated 71720 bytes for local input buffer
MERGE Data Link (25672): File mapped object /SHM_11011 accessed
MERGE Data Link (25672): Memory mapping 100669480 bytes
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000.
MERGE Master (25659): Link task has pid 25674
MERGE Data Link (25673): Message logger not contacted.
MERGE Data Link (25673): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25673): Configuration: index = 11, SHM key=11000, TCP port base = 11012, module = 10, hardware = 0
MERGE Data Link (25673): File mapped object /SHM_11000 accessed
MERGE Data Link (25673): Memory mapping 536628 bytes
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3cd54a000.
MERGE Data Link (25673): Allocated 71720 bytes for local input buffer
MERGE Data Link (25673): File mapped object /SHM_11012 accessed
MERGE Data Link (25673): Memory mapping 100669480 bytes
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Master (25659): Link task has pid 25675
MERGE Data Link (25674): Message logger not contacted.
MERGE Data Link (25674): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25674): Configuration: index = 12, SHM key=11000, TCP port base = 11013, module = 11, hardware = 0
MERGE Data Link (25674): File mapped object /SHM_11000 accessed
MERGE Data Link (25674): Memory mapping 536628 bytes
MERGE Data Link (25674): Shared memory segment located at address 0x7faec5dbd000.
MERGE Data Link (25674): Allocated 71720 bytes for local input buffer
MERGE Data Link (25674): File mapped object /SHM_11013 accessed
MERGE Data Link (25674): Memory mapping 100669480 bytes
MERGE Data Link (25674): Shared memory segment located at address 0x7faebfdbb000.
MERGE Master (25659): Link task has pid 25676
MERGE Master (25659): Link task has pid 25677
MERGE Data Link (25675): Message logger not contacted.
MERGE Data Link (25675): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25675): Configuration: index = 13, SHM key=11000, TCP port base = 11014, module = 12, hardware = 0
MERGE Data Link (25675): File mapped object /SHM_11000 accessed
MERGE Data Link (25675): Memory mapping 536628 bytes
MERGE Data Link (25675): Shared memory segment located at address 0x7fef43120000.
MERGE Data Link (25675): Allocated 71720 bytes for local input buffer
MERGE Data Link (25675): File mapped object /SHM_11014 accessed
MERGE Data Link (25675): Memory mapping 100669480 bytes
MERGE Data Link (25675): Shared memory segment located at address 0x7fef3d11e000.
MERGE Data Link (25676): Message logger not contacted.
MERGE Data Link (25676): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25676): Configuration: index = 14, SHM key=11000, TCP port base = 11015, module = 13, hardware = 0
MERGE Data Link (25676): File mapped object /SHM_11000 accessed
MERGE Data Link (25676): Memory mapping 536628 bytes
MERGE Data Link (25676): Shared memory segment located at address 0x7f6aea271000.
MERGE Data Link (25676): Allocated 71720 bytes for local input buffer
MERGE Data Link (25676): File mapped object /SHM_11015 accessed
MERGE Data Link (25676): Memory mapping 100669480 bytes
MERGE Data Link (25676): Shared memory segment located at address 0x7f6ae426f000.
MERGE Data Link (25677): Message logger not contacted.
MERGE Data Link (25677): MIDAS MERGE Data Link 64 bit Build May 11 2021  20:32:02
Creating NetVars
Creating NetVars complete
MERGE Data Link (25677): Configuration: index = 15, SHM key=11000, TCP port base = 11016, module = 14, hardware = 0
MERGE Data Link (25677): File mapped object /SHM_11000 accessed
MERGE Data Link (25677): Memory mapping 536628 bytes
MERGE Data Link (25677): Shared memory segment located at address 0x7fc4281e9000.
MERGE Data Link (25677): Allocated 71720 bytes for local input buffer
MERGE Data Link (25677): File mapped object /SHM_11016 accessed
MERGE Data Link (25677): Memory mapping 100669480 bytes
MERGE Data Link (25677): Shared memory segment located at address 0x7fc4221e7000.
MERGE Statistics/Rates (25661): MIDAS MERGE Statistics 64 bit Build May 11 2021  20:32:03
MERGE Statistics thread starting
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
MERGE Data Link (25665): Shared memory segment located at address 0x7f7412ae4000.
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Data Link (25663): Shared memory segment located at address 0x7f3be8601000.
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Data Link (25662): Shared memory segment located at address 0x7f1988e78000.
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Data Link (25667): Shared memory segment located at address 0x7f8b818c7000.
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25668): Shared memory segment located at address 0x7f88c1c98000.
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Data Link (25664): Shared memory segment located at address 0x7f0da12c6000.
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Data Link (25666): Shared memory segment located at address 0x7fb7eeec0000.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
MERGE Data Link (25670): Shared memory segment located at address 0x7ff76e4cc000.
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Data Link (25673): Shared memory segment located at address 0x7fe3c7548000.
MERGE Data Link (25672): Shared memory segment located at address 0x7f4bac87c000.
... 201 more lines ...
Attachment 3: 210602_TapeServerStartup
Tidy up
master(4596): Operation not permitted
Starting Tape Server

[1] 25639
MIDAS Tape Server: Message logger not contacted.
MIDAS Tape Server: MIDAS Tape Server Build Mar 16 2021  12:22:14
MIDAS Tape Server: Unable to change scheduling priority - Permission denied
MIDAS Tape Server: Using default startup
MIDAS Tape Server: Configuration: UDP port = 10205, SHM key=10205.
MIDAS Tape Server: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19721000.
MIDAS Tape Server: Configuration file used - /MIDAS/config/TS_10205/TS_configuration
MIDAS Tape Server: Stats task /MIDAS/TapeServer/Linux64/stats
MIDAS Tape Server: Using device file /dev/file/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device file /dev/file/1 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Using device sink /dev/null/0 /MIDAS/TapeServer/Linux64/driver
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 10305                 
MIDAS Tape Server: Data link /MIDAS/TapeServer/Linux64/linkTCP 21001 21002                
MIDAS Tape Server: Message reporting level = 0x180fff8
MIDAS Tape Server: Message logging level = 0xfff8
MIDAS Tape Server: Tape Server Options = 0x0
MIDAS Tape Server: File device path base = /TapeData
MIDAS Tape Server: Data buffer size = 65536
MIDAS Tape Server: Tape block size = 65536
MIDAS Tape Server: File mapped object /SHM_110205 of size 4195880 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19320000.
MIDAS Tape Server: File mapped object /SHM_210205 of size 3100 created
MIDAS Tape Server: Shared memory ID is 3
MIDAS Tape Server: Shared memory segment located at address 7fdf19881000.
MIDAS Tape Server: Capabilities restored.
MIDAS Tape Server: Master global area initialised.
MIDAS Tape Server: Stats task has pid 25641
MIDAS Tape Server: Driver process for /dev/file/0 has pid 25642
MIDAS Tape Server: Driver process for /dev/file/1 has pid 25643
MIDAS Tape Server: Driver process for /dev/null/0 has pid 25644
MIDAS Tape Server: Link task 0 has pid 25645
MIDAS Tape Server: Link task 1 has pid 25646
MIDAS Tape Server: Starting the RPC interface
MIDAS Tape Statistics: Message logger not contacted.
MIDAS Tape Statistics: MIDAS Tape Statistics Build Mar 16 2021  12:22:20
MIDAS Tape Statistics: Started with args 10205
MIDAS Tape Statistics: Configuration: SHM key=10205
MIDAS Tape Statistics: File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Statistics: Shared memory ID is 3
MIDAS Tape Statistics: Shared memory segment located at address 7f9c551f7000.
MIDAS Data Link (25645): Message logger not contacted.
MIDAS Data Link (25645): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Data Link (25645): Started with args 10205 10305                 
MIDAS Data Link (25645): Configuration: SHM key=10205, TCP port = 10305, Data Stream = 1
MIDAS Tape Driver (25644): Message logger not contacted.
MIDAS Tape Driver (25644): Message logger not contacted.
MIDAS Data Link (25646): Message logger not contacted.
MIDAS Tape Driver (25644): Started with args 2 10205
MIDAS Data Link (25645): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25645): Shared memory ID is 3
MIDAS Data Link (25645): Shared memory segment located at address 7f6ba30ef000.
MIDAS Data Link (25646): MIDAS Tape Data Link Build Mar 16 2021  12:22:19
MIDAS Tape Driver (25644): Configuration: driver=2, key=10205.
MIDAS Data Link (25646): Started with args 10205 21001 21002                
MIDAS Data Link (25645): Starting the network interface
MIDAS Tape Driver (25644): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25646): Configuration: SHM key=10205, TCP port = 21001, Data Stream = 2
MIDAS Tape Driver (25644): Shared memory ID is 3
MIDAS Tape Driver (25644): Shared memory segment located at address 7f09d718d000.
MIDAS Tape Driver (25644): Using device /dev/null/0 of type sink.
MIDAS Data Link (25646): File mapped object /SHM_10205 of size 1331104 created
MIDAS Data Link (25646): Shared memory ID is 3
MIDAS Data Link (25646): Shared memory segment located at address 7fa75ae3c000.
MIDAS Data Link (25646): Starting the network interface
MIDAS Tape Driver (25642): Message logger not contacted.
MIDAS Data Link (25645): TCP socket receive buffer was 87380 - now 249856
MIDAS Tape Driver (25642): Message logger not contacted.
MIDAS Tape Driver (25642): Started with args 0 10205
MIDAS Tape Driver (25642): Configuration: driver=0, key=10205.
MIDAS Tape Driver (25643): Message logger not contacted.
MIDAS Tape Driver (25642): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (25643): Message logger not contacted.
MIDAS Tape Driver (25642): Shared memory ID is 3
MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (25645): TCP socket send buffer was 16384 - now 249856
MIDAS Tape Driver (25643): Started with args 1 10205
MIDAS Tape Driver (25642): Shared memory segment located at address 7f48b1ea4000.
MIDAS Tape Driver (25642): Using device /dev/file/0 of type file.
MIDAS Data Link (25645): MIDAS Data Link thread 0 using TCP port 10305.
MIDAS Data Link (25645): Entering server loop
MIDAS Tape Driver (25643): Configuration: driver=1, key=10205.
MIDAS Data Link (25645): thread 0 listening on port 10305
MIDAS Tape Driver (25643): File mapped object /SHM_10205 of size 1331104 created
MIDAS Tape Driver (25643): Shared memory ID is 3
MIDAS Tape Driver (25643): Shared memory segment located at address 7f6044052000.
MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856
MIDAS Tape Driver (25643): Using device /dev/file/1 of type file.
MIDAS Data Link (25646): MIDAS Data Link thread 0 using TCP port 21001.
MIDAS Data Link (25646): TCP socket receive buffer was 87380 - now 249856
MIDAS Data Link (25646): TCP socket send buffer was 16384 - now 249856
MIDAS Data Link (25646): MIDAS Data Link thread 1 using TCP port 21002.
MIDAS Data Link (25646): Entering server loop
MIDAS Data Link (25646): thread 0 listening on port 21001
MIDAS Data Link (25646): thread 1 listening on port 21002
MIDAS Tape Server: Created RPC Program 28000205 Version 4 on UDP port 10205.
MIDAS Tape Server: Entering server loop
MIDAS Tape Server: MIDAS Tape Server now available on UDP port 10205.
MIDAS Data Link (25645): thread 0 accepted connection from 127.0.0.1, port 37330
MIDAS Data Link (25645): buffer size changed to 65536
MIDAS Tape Server: device /dev/file/0 allocated.
MIDAS Tape Server: Mounting volume S496 on device /dev/file/0.
MIDAS Tape Server: Opening file R31 on device /dev/file/0.
2222 64 0 31056336
3333 31056336 2048000 1 15 

Attachment 4: TS_configuration
#device configuration information
#    any line starting with a # is a comment and is ignored
stats /MIDAS/TapeServer/Linux64/stats
#    devices available   -   list ends with a null line
#    format       class_name     device_name    driver_task_path_name
#       class_name MUST be one of dlt, exabyte, scsitape, sink
file          /dev/file/0    /MIDAS/TapeServer/Linux64/driver
file          /dev/file/1    /MIDAS/TapeServer/Linux64/driver
sink          /dev/null/0     /MIDAS/TapeServer/Linux64/driver

#data link configuration information  - list end with a null line
#    format   link_task_path_name  
/MIDAS/TapeServer/Linux64/linkTCP 10305
/MIDAS/TapeServer/Linux64/linkTCP 21001 21002

#program options - list end with a null line
#tapeserver_options 0x2
tapeserver_options 0x0
msg_reporting_level 0x0180fff8
#   use 0x0080fff8 to enable msg logging
msg_logging_level 0xfff8
#    default for following is 16Kbytes - both MUST be the same at present
data_buffer_size 64
tape_block_size 64
disc_file_size 2000
#
file_path_base  /TapeData

#end of information


Attachment 5: CONTENTS
Index string MERGE.LinksAvailable&&RunNumber&&MERGE.LinksInUse&&MERGE.RunOptions
Stat.shift string 0
0x0006dead string 0x0000
MERGE.LinksAvailable string 16
Stat.offset string 0
RunNumber string 0
Rate.channels string 1024
Stat.channels string 1024
MERGE.RunOptions string 3
MERGE.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%1%
Merge.LinksInUse string 1%1%1%1%1%1%1%1%1%1%1%1%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%0%
Attachment 6: 210602_merger.png
210602_merger.png
  352   Sun May 30 08:57:53 2021 TDSunday 30 May
09.55 DAQ continues - file S496/R18_588
      alpha background

      increased data rate due to hot HEC channels since c. 06.00 this am - ASIC check fixes problem

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b7 : 	 7  
aida02 fault 	 0xfe72 : 	 0xfe79 : 	 7  
aida03 fault 	 0x97f4 : 	 0x97fb : 	 7  
aida04 fault 	 0x950 : 	 0x957 : 	 7  
aida05 fault 	 0x8e93 : 	 0x8e98 : 	 5  
aida06 fault 	 0x545a : 	 0x545f : 	 5  
aida07 fault 	 0xc04d : 	 0xc052 : 	 5  
aida08 fault 	 0xa0f7 : 	 0xa0fc : 	 5  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      6      9      5      1      3      2      2      2      2      4      6   : 37008
aida02 :      3      5      5      2      2      4      3      3      2      4      6   : 37956
aida03 :     22     12      7      5      3      3      2      3      3      3      6   : 36872
aida04 :     40     13      5      5      3      4      2      3      3      3      6   : 37048
aida05 :     15      5      5      4      2      2      2      2      2      4      6   : 37044
aida06 :     11      6      2      1      2      3      3      4      2      4      6   : 38300
aida07 :     17      5      4      3      1      3      3      2      2      4      6   : 37324
aida08 :      1      8      1      2      2      4      4      3      2      4      6   : 38164
aida09 :     10      8      6      1      2      4      3      3      1      4      6   : 36968
aida10 :     14      7      5      2      1      3      3      3      1      4      6   : 36800
aida11 :      7      9      3      1      2      3      3      3      1      4      6   : 36788
aida12 :      3      5      2      3      1      4      2      3      2      4      6   : 37620
aida13 :     12      4      4      2      1      3      2      2      2      4      6   : 37008
aida14 :     11     10      5      1      1      3      2      2      2      4      6   : 37036
aida15 :     11      5      1      2      1      3      3      4      2      4      6   : 38244
aida16 :     10      4      8      1      2      3      2      2      2      4      6   : 37096

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.24 All histograms, statistics and merger statistics zero'd

16.16 analysis of file S496/R18_541 - attachment 9
       max deadtime 0.02% aida03

18.40 DAQ continues - file S496/R18_589
      alpha background

      increased data rate due to hot HEC channels since c. 18.00 this am - ASIC check fixes problem
Attachment 1: Screenshot_2021-05-30_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-30_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-30_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-30_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-30_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-30_Statistics_aidas-gsi.png
Screenshot_2021-05-30_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-30_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-30_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 960.png
960.png
Attachment 8: 961.png
961.png
Attachment 9: R18_589
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      31985
          ADC data format:   58630511 (   25364.6 Hz)
        Other data format:  203166719 (   87893.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         35 (       0.0 Hz)
                               RESUME:         35 (       0.0 Hz)
                              SYNC100:      40596 (      17.6 Hz)
                              WR48-63:      40596 (      17.6 Hz)
                           FEE64 disc:    1341777 (     580.5 Hz)
                             MBS info:  201743680 (   87277.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2311.511 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             1879.226
  2                0.365                0.000
  3                0.313                0.000
  4                0.000             1697.189
  5                0.000              557.330
  6                0.000             1686.560
  7                0.000              647.171
  8                0.000                0.000
  9                0.000                0.000
 10                0.382                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        157          0          0          0          0          0          0          0          0          0          0          0
  1    3314609       1296          0          0          0          0        648        648          0          0          0          0
  2        413   87031526          0          0         19         19      14856      14856          0   87001776          0          0
  3   42258127   88383449          0          0         15         15      18954      18954    1341775   87003736          0          0
  4        153    6936262          0          0          0          0        860        860          0    6934542          0          0
  5    6312304    6939162          0          0          0          0       2310       2310          0    6934542          0          0
  6        544    6936170          0          0          0          0        814        814          0    6934542          0          0
  7    6741331    6938850          0          0          0          0       2154       2154          0    6934542          0          0
  8        448          0          0          0          0          0          0          0          0          0          0          0
  9         79          0          0          0          0          0          0          0          0          0          0          0
 10       1189          2          0          0          1          1          0          0          0          0          0          0
 11        111          2          0          0          0          0          0          0          2          0          0          0
 12        767          0          0          0          0          0          0          0          0          0          0          0
 13         99          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        180          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.504s ( 1015.271 blocks/s,  63.454 Mb/s)
  351   Sat May 29 13:13:45 2021 TDSaturday 29 May
14.10 DAQ continues - file S496/R18_539
      alpha background

      increased data rate due to ht HEC channels since c. 06.00 this am - ASIC check fixes problem

      All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b7 : 	 7  
aida02 fault 	 0xfe72 : 	 0xfe79 : 	 7  
aida03 fault 	 0x97f4 : 	 0x97fb : 	 7  
aida04 fault 	 0x950 : 	 0x957 : 	 7  
aida05 fault 	 0x8e93 : 	 0x8e97 : 	 4  
aida06 fault 	 0x545a : 	 0x545e : 	 4  
aida07 fault 	 0xc04d : 	 0xc051 : 	 4  
aida08 fault 	 0xa0f7 : 	 0xa0fb : 	 4  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      8      7      2      2      2      2      2      2      4      6   : 37036
aida02 :     14      3      5      2      2      4      3      3      2      4      6   : 37984
aida03 :     15     11      8      6      3      3      2      3      3      3      6   : 36884
aida04 :     43     13      5      5      3      4      2      3      3      3      6   : 37060
aida05 :     19      3      5      4      2      2      2      2      2      4      6   : 37044
aida06 :     17      4      1      3      2      3      3      4      2      4      6   : 38356
aida07 :     16      6      2      1      0      2      2      3      2      4      6   : 37296
aida08 :     21      8      1      1      3      4      4      3      2      4      6   : 38276
aida09 :     13     10      2      1      2      3      2      2      2      4      6   : 37060
aida10 :     13      9      4      1      1      3      3      3      1      4      6   : 36764
aida11 :     13      9      2      1      2      3      3      3      1      4      6   : 36796
aida12 :     11      6      4      2      2      4      2      3      2      4      6   : 37724
aida13 :      7      6      6      2      1      3      2      2      2      4      6   : 37036
aida14 :      9      7      7      1      1      3      2      2      2      4      6   : 37036
aida15 :     14      5      2      2      1      3      3      4      2      4      6   : 38272
aida16 :      9      4     10      1      2      3      2      2      2      4      6   : 37124

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.24 All histograms, statistics and merger statistics zero'd

16.16 analysis of file S496/R18_541 - attachment 9
       max deadtime 0.12% aida04

18.40 DAQ continues - file S496/R18_551
      alpha background

      increased data rate due to ht HEC channels since c. 18.00 this am - ASIC check fixes problem
Attachment 1: Screenshot_2021-05-29_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-29_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-29_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-29_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-29_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-29_Statistics_aidas-gsi.png
Screenshot_2021-05-29_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-29_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-29_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 950.png
950.png
Attachment 8: 951.png
951.png
Attachment 9: R18_541
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   46792076 (   18702.9 Hz)
        Other data format:  215127929 (   85987.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         89 (       0.0 Hz)
                               RESUME:         89 (       0.0 Hz)
                              SYNC100:      41318 (      16.5 Hz)
                              WR48-63:      41318 (      16.5 Hz)
                           FEE64 disc:    1013016 (     404.9 Hz)
                             MBS info:  214032099 (   85549.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2501.862 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             1964.282
  2                0.467                0.000
  3                3.050                0.000
  4                0.000             1832.921
  5                0.000              844.521
  6                0.000             1829.325
  7                0.000              869.373
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        499          0          0          0          0          0          0          0          0          0          0          0
  1      74891         24          0          0          0          0         12         12          0          0          0          0
  2       1032   92085039          0          0         27         27      16251      16251          0   92052483          0          0
  3   37271457   93008741          0          0         62         62      19135      19135    1013015   91957332          0          0
  4        552    7507421          0          0          0          0        925        925          0    7505571          0          0
  5    4051453    7509613          0          0          0          0       2021       2021          0    7505571          0          0
  6       1401    7507471          0          0          0          0        950        950          0    7505571          0          0
  7    5387708    7509619          0          0          0          0       2024       2024          0    7505571          0          0
  8        460          0          0          0          0          0          0          0          0          0          0          0
  9        183          0          0          0          0          0          0          0          0          0          0          0
 10        537          0          0          0          0          0          0          0          0          0          0          0
 11        388          0          0          0          0          0          0          0          0          0          0          0
 12        674          0          0          0          0          0          0          0          0          0          0          0
 13        244          1          0          0          0          0          0          0          1          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        597          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.855s ( 1004.537 blocks/s,  62.784 Mb/s)
  350   Fri May 28 10:29:07 2021 TDFriday 28 May
11.30 DAQ continues - file S496/R18_472
      alpha background

      All system wide checks OK *except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b2 : 	 2  
aida02 fault 	 0xfe72 : 	 0xfe74 : 	 2  
aida03 fault 	 0x97f4 : 	 0x97f6 : 	 2  
aida04 fault 	 0x950 : 	 0x952 : 	 2  
aida05 fault 	 0x8e93 : 	 0x8e95 : 	 2  
aida06 fault 	 0x545a : 	 0x545c : 	 2  
aida07 fault 	 0xc04d : 	 0xc04f : 	 2  
aida08 fault 	 0xa0f7 : 	 0xa0f9 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      4      7      1      3      2      2      2      2      4      6   : 36980
aida02 :      3      1      5      1      2      4      3      3      2      4      6   : 37892
aida03 :      8      8      6      6      3      3      2      3      3      3      6   : 36800
aida04 :     37     12      5      6      3      4      2      3      3      3      6   : 37060
aida05 :     11      5      6      4      2      2      2      2      2      4      6   : 37044
aida06 :      4      3      3      3      2      2      2      3      3      4      6   : 38456
aida07 :      1     12      3      2      0      3      3      2      2      4      6   : 37204
aida08 :     13      8      1      2      3      4      3      3      2      4      6   : 38020
aida09 :      7      7      5      1      1      3      2      2      2      4      6   : 36996
aida10 :     12      7      9      2      1      3      3      3      1      4      6   : 36856
aida11 :      2      9      3      1      2      3      3      3      1      4      6   : 36768
aida12 :     11      4      3      3      2      4      2      3      2      4      6   : 37724
aida13 :     10      8      6      2      2      3      2      2      2      4      6   : 37128
aida14 :     16      7      7      1      0      4      2      2      2      4      6   : 37128
aida15 :     12      6      2      2      1      3      3      4      2      4      6   : 38272
aida16 :      9      4     10      1      2      3      2      2      2      4      6   : 37124

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_471 - attachment 9
       max deadtime 0.18% aida04
Attachment 1: Screenshot_2021-05-28_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-28_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-28_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-28_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-28_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-28_Statistics_aidas-gsi.png
Screenshot_2021-05-28_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-28_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-28_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 940.png
940.png
Attachment 8: 941.png
941.png
Attachment 9: R18_471
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   57407666 (   20753.5 Hz)
        Other data format:  204512336 (   73933.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        116 (       0.0 Hz)
                               RESUME:        116 (       0.0 Hz)
                              SYNC100:      42303 (      15.3 Hz)
                              WR48-63:      42303 (      15.3 Hz)
                           FEE64 disc:    1160567 (     419.6 Hz)
                             MBS info:  203266931 (   73483.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        109 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2766.172 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2700.364
  2                0.332                0.000
  3                4.856                0.000
  4                0.000                0.000
  5                0.000             1685.309
  6                0.000                0.000
  7                0.000             1567.327
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        181         23          0          0          0          0          0          0         23          0          0         13
  1     838470        386          0          0          0          0        166        166         54          0          0          0
  2        162  101753181          0          0         19         19      17931      17931         15  101717266          0          5
  3   42392398  102752755          0          0         97         97      21331      21331    1160234  101549665          0         12
  4        211         21          0          0          0          0          0          0         21          0          0         10
  5    6478076       2740          0          0          0          0       1337       1337         66          0          0          0
  6        260         17          0          0          0          0          0          0         17          0          0          4
  7    7695649       3163          0          0          0          0       1537       1537         89          0          0         28
  8        132         14          0          0          0          0          1          1         12          0          0         10
  9        368          5          0          0          0          0          0          0          5          0          0          4
 10        173         18          0          0          0          0          0          0         18          0          0          5
 11        303          5          0          0          0          0          0          0          5          0          0          5
 12        186          1          0          0          0          0          0          0          1          0          0          7
 13        585          2          0          0          0          0          0          0          2          0          0          2
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        512          5          0          0          0          0          0          0          5          0          0          4
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.742s ( 1008.122 blocks/s,  63.008 Mb/s)
  349   Thu May 27 12:25:53 2021 TDThursday 27 May
13.29 DAQ continues - file S496/R18_443
      alpha background

      All system wide checks OK *except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b2 : 	 2  
aida02 fault 	 0xfe72 : 	 0xfe74 : 	 2  
aida03 fault 	 0x97f4 : 	 0x97f6 : 	 2  
aida04 fault 	 0x950 : 	 0x952 : 	 2  
aida05 fault 	 0x8e93 : 	 0x8e95 : 	 2  
aida06 fault 	 0x545a : 	 0x545c : 	 2  
aida07 fault 	 0xc04d : 	 0xc04f : 	 2  
aida08 fault 	 0xa0f7 : 	 0xa0f9 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      7      7      1      3      2      2      2      2      4      6   : 37008
aida02 :      3      1      5      1      3      4      3      3      2      4      6   : 37956
aida03 :      2      6      5      6      2      4      2      3      3      3      6   : 36808
aida04 :     24     12      5      5      4      3      2      3      3      3      6   : 36912
aida05 :      8      1      9      3      2      2      2      2      2      4      6   : 37016
aida06 :      3      6      2      2      2      2      2      3      3      4      6   : 38428
aida07 :      6     17      1      2      0      3      3      2      2      4      6   : 37232
aida08 :      4      2      1      2      3      4      3      3      2      4      6   : 37936
aida09 :     15      7      5      2      3      4      3      3      1      4      6   : 37060
aida10 :      7      6      9      2      0      3      3      3      1      4      6   : 36764
aida11 :     19     10      2      0      2      3      3      3      1      4      6   : 36796
aida12 :     11      6      4      2      2      4      2      3      2      4      6   : 37724
aida13 :     17      9      2      0      0      4      2      2      2      4      6   : 37036
aida14 :     15      4      7      1      1      3      2      2      2      4      6   : 37036
aida15 :      9      6      1      2      1      3      3      4      2      4      6   : 38244
aida16 :     11     11      8      2      0      3      2      2      2      4      6   : 37060

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_442 - attachment 9
       max deadtime 0.28% aida04
Attachment 1: Screenshot_2021-05-27_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-27_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-27_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-27_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-27_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-27_Statistics_aidas-gsi.png
Screenshot_2021-05-27_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-27_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-27_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 930.png
930.png
Attachment 8: 931.png
931.png
Attachment 9: R18_442
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   68693190 (   21525.3 Hz)
        Other data format:  193226813 (   60548.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        114 (       0.0 Hz)
                               RESUME:        114 (       0.0 Hz)
                              SYNC100:      43889 (      13.8 Hz)
                              WR48-63:      43889 (      13.8 Hz)
                           FEE64 disc:    1335474 (     418.5 Hz)
                             MBS info:  191803333 (   60102.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          3 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     3191.275 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2989.890
  2                0.511              643.077
  3                8.760                0.000
  4                0.000                0.000
  5                0.000             1578.752
  6                0.000                0.000
  7                0.000             1340.150
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        155          0          0          0          0          0          0          0          0          0          0          0
  1     527962        216          0          0          0          0        108        108          0          0          0          0
  2        533   96091128          0          0         28         28      16832      16832          0   96057408          0          0
  3   49735236   97127126          0          0         86         86      22778      22778    1335473   95745925          0          0
  4        118          2          0          0          0          0          1          1          0          0          0          0
  5    8268444       3874          0          0          0          0       1937       1937          0          0          0          0
  6        632          0          0          0          0          0          0          0          0          0          0          0
  7   10156762       4466          0          0          0          0       2233       2233          0          0          0          0
  8        500          0          0          0          0          0          0          0          0          0          0          0
  9        102          0          0          0          0          0          0          0          0          0          0          0
 10       1498          0          0          0          0          0          0          0          0          0          0          0
 11        105          0          0          0          0          0          0          0          0          0          0          0
 12        886          0          0          0          0          0          0          0          0          0          0          0
 13         95          1          0          0          0          0          0          0          1          0          0          2
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        162          0          0          0          0          0          0          0          0          0          0          1
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.027s (  999.146 blocks/s,  62.447 Mb/s)
  348   Wed May 26 13:21:59 2021 TDWednesday 26 May
14.22 DAQ continues - file S496/R18_369
      alpha background

      All system wide checks OK *except*

	
		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     12      4      6      1      2      3      2      2      2      4      6   : 37072
aida02 :     12      2      4      1      3      4      3      3      2      4      6   : 37984
aida03 :     12      6      7      6      2      4      3      2      3      3      6   : 36624
aida04 :     36     11      4      6      4      3      2      3      3      3      6   : 36968
aida05 :      2      4      6      3      2      2      2      2      2      4      6   : 36968
aida06 :      8      5      3      2      2      2      2      3      3      4      6   : 38456
aida07 :     10      4      6      4      3      3      3      3      2      4      6   : 37992
aida08 :     21      8      3      2      3      3      3      3      2      4      6   : 37956
aida09 :     17     10      3      2      3      4      3      3      1      4      6   : 37060
aida10 :      3      6     10      2      1      3      3      3      1      4      6   : 36828
aida11 :     12      6      4      2      2      3      3      3      1      4      6   : 36832
aida12 :      2      9      5      1      3      3      2      3      2      4      6   : 37632
aida13 :      4      7      2      1      3      3      3      3      2      4      6   : 37832
aida14 :     15      6      6      1      0      4      2      2      2      4      6   : 37100
aida15 :      5      4      3      2      1      3      3      4      2      4      6   : 38244
aida16 :     17      6      6      2      1      3      2      2      2      4      6   : 37076

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
      Lost activity monitor - attachment 2
      1.8.W spectra - attachments 3 & 4
      ADC data item stats - attachment 5
      FEE64 temps OK - attachment 6
      DSSSD bias & leakage current OK - attachment 7
      Merger/Tape Server/Merger stats - attachment 8
       no merger errors reported since previous restart

14.30 analysis of file S496/R18_340 - attachment 9
       max deadtime 0.31% aida04
Attachment 1: Screenshot_2021-05-26_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-26_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-26_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-26_Lost_activity_monitor_aidas-gsi.png
Attachment 3: Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-26_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-26_Statistics_aidas-gsi.png
Screenshot_2021-05-26_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-05-26_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-26_Temperature_and_status_scan_aidas-gsi.png
Attachment 7: 920.png
920.png
Attachment 8: 921.png
921.png
Attachment 9: R18_368
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   60865731 (   22299.7 Hz)
        Other data format:  201054280 (   73661.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        152 (       0.1 Hz)
                               RESUME:        152 (       0.1 Hz)
                              SYNC100:      42166 (      15.4 Hz)
                              WR48-63:      42166 (      15.4 Hz)
                           FEE64 disc:    1309767 (     479.9 Hz)
                             MBS info:  199659877 (   73150.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:        107 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2729.447 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2401.079
  2                0.453                0.000
  3                8.570                0.000
  4                0.000                0.000
  5                0.000             1507.937
  6                0.000                0.000
  7                0.000             1499.170
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0       1115         29          0          0          0          0          1          1         27          0          0         10
  1    2766811      11645          0          0          0          0        574        574      10497          0          0         18
  2       2194  100014163          0          0         27         27      17354      17354          0   99979401          0          1
  3   44009553  100997602          0          0        125        125      21286      21286    1274310   99680470          0          5
  4        902          6          0          0          0          0          0          0          0          6          0          0
  5    6872755      14651          0          0          0          0       1451       1451      11749          0          0         10
  6       1948         13          0          0          0          0          1          1         11          0          0          9
  7    7202790      16106          0          0          0          0       1497       1497      13112          0          0          9
  8        531          0          0          0          0          0          0          0          0          0          0          0
  9        394          0          0          0          0          0          0          0          0          0          0          3
 10       1118          1          0          0          0          0          0          0          1          0          0          0
 11        900         26          0          0          0          0          0          0         26          0          0         12
 12       1756         13          0          0          0          0          0          0         13          0          0          8
 13       1346         11          0          0          0          0          1          1          9          0          0          7
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15       1618         14          0          0          0          0          1          1         12          0          0         15
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   31.695s ( 1009.613 blocks/s,  63.101 Mb/s)
  347   Tue May 25 11:16:54 2021 TDTuesday 25 May
12.16 DAQ continues file S496/R18_324
      alpha background

      Tape server c. 2.5Mb/s
      ASIC check
      Tape server c. 300kb/s - presumably hot HEC channels

12.20 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      6      2      1      3      2      2      2      4      6   : 37016
aida02 :      3      6      5      3      3      4      3      2      1      3      7   : 38572
aida03 :     21     10      8      6      3      3      2      3      2      4      6   : 37924
aida04 :     29     15      4      5      3      4      2      3      3      3      6   : 37004
aida05 :     17      5      2      2      2      2      2      2      2      4      6   : 36940
aida06 :      1      6      0      1      2      2      2      3      3      4      6   : 38356
aida07 :     11      6      7      4      3      3      3      3      2      4      6   : 38028
aida08 :     13      8      3      2      2      3      3      2      1      3      7   : 38372
aida09 :      7      9      2      1      3      3      3      3      2      4      6   : 37860
aida10 :      9      8      4      2      0      3      3      3      1      4      6   : 36708
aida11 :     16      6      3      2      2      3      3      3      1      4      6   : 36832
aida12 :     21     10      7      4      2      4      2      2      1      3      7   : 38420
aida13 :      8      3      3      1      3      3      3      3      2      4      6   : 37832
aida14 :     14      3      6      1      0      4      2      2      2      4      6   : 37072
aida15 :      2      4      2      2      1      3      3      4      2      4      6   : 38216
aida16 :     16      5      5      3      0      3      2      2      2      4      6   : 37016


Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage cuuent - most recent 7 days - attachment 1
      Rate spectra - attachment 2
      ADC data item stats - attachment 3
      FEE64 temps OK - attachment 4
      DSSSD bias & leakage current OK - attachment 5
      Merger/Tape Server/Merger stats - attachment 6
       no merger errors reported since previous restart

17.30 analysis of file S496/R18_340
       max deadtime 0.36% aida04
Attachment 1: Screenshot_2021-05-25_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-25_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-25_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-25_Spectrum_Browser_aidas-gsi.png
Attachment 3: Screenshot_2021-05-25_Statistics_aidas-gsi.png
Screenshot_2021-05-25_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-05-25_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-25_Temperature_and_status_scan_aidas-gsi.png
Attachment 5: 910.png
910.png
Attachment 6: 911.png
911.png
Attachment 7: R18_340
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   63855540 (   23748.1 Hz)
        Other data format:  198064462 (   73660.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        109 (       0.0 Hz)
                               RESUME:        109 (       0.0 Hz)
                              SYNC100:      42015 (      15.6 Hz)
                              WR48-63:      42015 (      15.6 Hz)
                           FEE64 disc:    1291036 (     480.1 Hz)
                             MBS info:  196689178 (   73149.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:         35 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2688.871 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2304.866
  2                0.344                0.000
  3                9.625                0.000
  4                0.000                0.000
  5                0.000             1450.665
  6                0.000                0.000
  7                0.000             1243.447
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        303          7          0          0          0          0          1          1          5          0          0          5
  1    3031185       1240          0          0          0          0        620        620          0          0          0          0
  2        536   98548687          0          0         20         20      16872      16872          2   98514901          0          3
  3   44821969   99507955          0          0         89         89      21248      21248    1291004   98174277          0          0
  4        514          3          0          0          0          0          1          1          1          0          0          1
  5    7529304       3056          0          0          0          0       1528       1528          0          0          0          0
  6       1314          2          0          0          0          0          0          0          2          0          0          2
  7    8467764       3488          0          0          0          0       1744       1744          0          0          0          0
  8        226          4          0          0          0          0          0          0          4          0          0          4
  9        144          4          0          0          0          0          0          0          4          0          0          4
 10        524          1          0          0          0          0          0          0          1          0          0          0
 11        432          2          0          0          0          0          0          0          2          0          0          2
 12        719          0          0          0          0          0          0          0          0          0          0          0
 13        154          7          0          0          0          0          1          1          5          0          0          7
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        452          6          0          0          0          0          0          0          6          0          0          7
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   39.492s (  810.287 blocks/s,  50.643 Mb/s)
  346   Mon May 24 09:57:07 2021 TDMonday 24 May
07.30 Hot HEC channels multiple FEE64s
      ASIC check fixes

10.56 DAQ continues OK file S496/R18_75

      System wide checks OK * except*

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	 Base 		Current 	Difference
aida01 fault 	 0x82b0 : 	 0x82b1 : 	 1  
aida02 fault 	 0xfe72 : 	 0xfe73 : 	 1  
aida03 fault 	 0x97f4 : 	 0x97f5 : 	 1  
aida04 fault 	 0x950 : 	 0x951 : 	 1  
aida05 fault 	 0x8e93 : 	 0x8e94 : 	 1  
aida06 fault 	 0x545a : 	 0x545b : 	 1  
aida07 fault 	 0xc04d : 	 0xc04e : 	 1  
aida08 fault 	 0xa0f7 : 	 0xa0f8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      2      7      2      1      4      3      2      1      4      7   : 40452
aida02 :     13      3      3      1      3      4      2      2      2      4      7   : 41308
aida03 :     17      6      2      2      1      4      2      3      2      3      7   : 39700
aida04 :      8      6      1      0      1      3      1      4      2      3      7   : 39712
aida05 :     10      3      5      3      4      3      3      2      1      4      7   : 40560
aida06 :      4      4      2      0      2      3      2      3      3      4      7   : 42576
aida07 :      5      3      2      3      1      3      1      3      2      4      7   : 41324
aida08 :     11      5      1      1      2      3      3      2      2      4      7   : 41348
aida09 :      7      4      3      0      2      3      1      3      2      4      7   : 41324
aida10 :     11      4      2      2      2      3      2      4      1      4      7   : 41132
aida11 :     18      5      0      2      2      4      2      2      1      4      7   : 40240
aida12 :      2      3      0      1      2      3      2      2      2      4      7   : 41024
aida13 :      4      5      1      2      2      3      2      4      1      4      7   : 41096
aida14 :      9      3      1      2      1      4      2      2      2      4      7   : 41164
aida15 :      2      3      1      3      2      2      3      3      2      4      7   : 41744
aida16 :      5      5      2      0      3      3      2      4      1      4      7   : 41116

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 12:04:25 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      Grafana - DSSSD bias & leakage current - most recent 7 days - attachment 1
      Statistics - disc, pause, WR 28-47, correlation scaler - attachments 2-5
      1.8.H spectra - attachments 6-8
      1.8.L spectra - attachments 9 & 10
      Stat spectra - attachments 11 & 12
      1.8.W spectra - 20us FSR - attachments 13 & 14
      ADC data items - attachment 15
      FEE64 temperatures OK - attachment 16
      DSSSD bias & leakage currents OK - attachment 17
      Merger/Tape Server/Merger statistics - attachment 18
       no merger errors reported since previous restart

      analysis of file R18_74
       max dead time 0.05%

10.17 ASIC check
      All histograms, statistics, merger statistics zero'd
Attachment 1: Screenshot_2021-05-24_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-24_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-24_Statistics_aidas-gsi(4).png
Screenshot_2021-05-24_Statistics_aidas-gsi(4).png
Attachment 3: Screenshot_2021-05-24_Statistics_aidas-gsi(3).png
Screenshot_2021-05-24_Statistics_aidas-gsi(3).png
Attachment 4: Screenshot_2021-05-24_Statistics_aidas-gsi(2).png
Screenshot_2021-05-24_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-24_Statistics_aidas-gsi(1).png
Screenshot_2021-05-24_Statistics_aidas-gsi(1).png
Attachment 6: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(8).png
Attachment 7: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(7).png
Attachment 8: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(6).png
Attachment 9: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(5).png
Attachment 10: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(4).png
Attachment 11: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(3).png
Attachment 12: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(2).png
Attachment 13: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi(1).png
Attachment 14: Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-24_Spectrum_Browser_aidas-gsi.png
Attachment 15: Screenshot_2021-05-24_Statistics_aidas-gsi.png
Screenshot_2021-05-24_Statistics_aidas-gsi.png
Attachment 16: Screenshot_2021-05-24_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-24_Temperature_and_status_scan_aidas-gsi.png
Attachment 17: 900.png
900.png
Attachment 18: 901.png
901.png
Attachment 19: R18_74
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   42982076 (   14466.2 Hz)
        Other data format:  218937925 (   73686.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         59 (       0.0 Hz)
                               RESUME:         59 (       0.0 Hz)
                              SYNC100:      43069 (      14.5 Hz)
                              WR48-63:      43069 (      14.5 Hz)
                           FEE64 disc:    1163669 (     391.6 Hz)
                             MBS info:  217688000 (   73265.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     2971.209 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.425                0.000
  3                1.488                0.000
  4                0.000                0.000
  5                0.000             2622.333
  6                0.000                0.000
  7                0.000             2340.476
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0         64          0          0          0          0          0          0          0          0          0          0          0
  1       2227          3          0          0          0          0          1          1          1          0          0          0
  2         75  108903724          0          0         25         25      19964      19964          0  108863746          0          0
  3   35357926  110031040          0          0         34         34      21525      21525    1163668  108824254          0          0
  4         89          0          0          0          0          0          0          0          0          0          0          0
  5    3145189       1316          0          0          0          0        658        658          0          0          0          0
  6         64          0          0          0          0          0          0          0          0          0          0          0
  7    4475895       1842          0          0          0          0        921        921          0          0          0          0
  8         56          0          0          0          0          0          0          0          0          0          0          0
  9         67          0          0          0          0          0          0          0          0          0          0          0
 10        115          0          0          0          0          0          0          0          0          0          0          0
 11         65          0          0          0          0          0          0          0          0          0          0          0
 12         65          0          0          0          0          0          0          0          0          0          0          0
 13         77          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15        102          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   40.766s (  784.975 blocks/s,  49.061 Mb/s)
  345   Sat May 22 23:40:32 2021 TDSunday 23 May
00.40 Unable to connect to FEE64s 1-7
      DAQ power cycled OK

      ASIC settings 2021Apr29-13-16-00
      slow comparator 0x64 *all* FEE64s

      BNC PB-5 OFF

      file S496/R16

      alpha background

07.17 Some hot HEC channels
      ASIC check fixes issue
      System wide checks OK *except*

		 Base 		Current 	Difference
aida06 fault 	 0x5457 : 	 0x5457 : 	 0  
aida06 : WR status 0x10
 White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1cd : 	 461  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      3      1      4      3      2      1      4      7   : 40552
aida02 :      3      2      2      1      2      3      3      2      2      4      7   : 41308
aida03 :     14      5      0      1      1      3      2      4      2      3      7   : 40000
aida04 :     14      4      5      1      3      3      1      4      2      3      7   : 39944
aida05 :     12      4      1      3      2      3      3      2      2      4      7   : 41408
aida06 :      8      3      2      1      2      4      2      3      3      4      7   : 42744
aida07 :     10      4      3      2      1      2      2      3      2      4      7   : 41464
aida08 :      7      5      2      0      3      3      3      2      2      4      7   : 41380
aida09 :     10      5      1      1      1      2      2      3      2      4      7   : 41408
aida10 :     11      6      3      2      1      3      2      2      2      4      7   : 41100
aida11 :     12      8      3      2      2      2      3      2      1      4      7   : 40288
aida12 :      9      9      3      1      2      3      3      2      1      4      7   : 40380
aida13 :     10      4      0      1      1      2      1      3      2      4      7   : 41128
aida14 :      9      4      2      1      1      2      1      3      2      4      7   : 41156
aida15 :      5      4      0      3      3      3      3      3      2      4      7   : 41940
aida16 :      9      6      3      2      0      2      1      3      2      4      7   : 41156

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 23 00:29:48 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

      FEE64 temperatures OK - attachment 1
      ADC data item stats - attachment 2
      Grafana DSSSD bias & leakage currents most recent 7 days - attachment 3
      DSSSD bias & leakage currents OK - attachment 4

11.18 Analysis of file S496/R16_33 - attachment 5

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  685M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.0T  867G  88% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

867/2 x 3279s = 16.5 days to filesystem full - OK

12.11 aida15 zero data/poll stats - re-loaded DAQ program OK
      aida04 unable to connect - multiple merger starts ineffective - aida04 rebooted and DAQ re-started OK

      DAQ continues file S496/R18
Attachment 1: Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-23_AIDA_Alerting_-_Grafana.png
Attachment 2: Screenshot_2021-05-23_Statistics_aidas-gsi.png
Screenshot_2021-05-23_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: 810.png
810.png
Attachment 5: R16_33
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   47385414 (   14450.2 Hz)
        Other data format:  214534594 (   65422.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         61 (       0.0 Hz)
                               RESUME:         61 (       0.0 Hz)
                              SYNC100:      44215 (      13.5 Hz)
                              WR48-63:      44215 (      13.5 Hz)
                           FEE64 disc:    1311957 (     400.1 Hz)
                             MBS info:  213134085 (   64995.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     3279.230 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.386                0.000
  3                1.000                0.000
  4                0.000                0.000
  5                0.000             2849.824
  6                0.000                0.000
  7                0.000             2608.689
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0         71          0          0          0          0          0          0          0          0          0          0          0
  1       1455          0          0          0          0          0          0          0          0          0          0          0
  2         87  106617422          0          0         35         35      20186      20186          0  106576980          0          0
  3   39184250  107913590          0          0         26         26      22238      22238    1311957  106557105          0          0
  4         63          0          0          0          0          0          0          0          0          0          0          0
  5    3345685       1512          0          0          0          0        756        756          0          0          0          0
  6         60          0          0          0          0          0          0          0          0          0          0          0
  7    4853274       2070          0          0          0          0       1035       1035          0          0          0          0
  8         69          0          0          0          0          0          0          0          0          0          0          0
  9         45          0          0          0          0          0          0          0          0          0          0          0
 10         70          0          0          0          0          0          0          0          0          0          0          0
 11         68          0          0          0          0          0          0          0          0          0          0          0
 12         89          0          0          0          0          0          0          0          0          0          0          0
 13         57          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15         71          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   44.293s (  722.462 blocks/s,  45.154 Mb/s)
  344   Sat May 22 13:40:56 2021 TDSaturday 22 May
14.37 DAQ continues file S496/R15_14

      Grafana - DSSSD bias & leakage current - attachment 1
      1.8.L spectra - attachments 2 & 3
      Stat spectra - attachments 4 & 5
      FEE64 temperatures OK - attachment 6
      stats - ADC data items, pause items OK - attachments 7 & 8
      DSSSD bias & leakage current OK - attachment 9
      Merger/Tape Server/Merger stats - attachment 10
      analysis of file S496/R14_1920 - attachment 11
       all FEE64 deadtime 0% as expected

17.30 aida01, 3, 5, 9 & 11 have hot HEC channels
       ASIC check fixes
      All histogram, statistics & merger statistics zero'd

      DAQ continues
      Note gamma-ray source tests have started with FATIMA and DEGAS

20.20 Analysis file R15_19 - attachment 12

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  665M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.3T  530G  93% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

        estimate 90/2 x 3168s = 40h to filesystem full - OK

20.22 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0xf82c : 	 0xf82f : 	 3  
aida02 fault 	 0xdda6 : 	 0xdda9 : 	 3  
aida03 fault 	 0xb13d : 	 0xb140 : 	 3  
aida04 fault 	 0xf4cd : 	 0xf4d0 : 	 3  
aida05 fault 	 0x500 : 	 0x575 : 	 117  
aida06 fault 	 0xc6e7 : 	 0xc6e9 : 	 2  
aida07 fault 	 0x4920 : 	 0x4922 : 	 2  
aida08 fault 	 0xbd9a : 	 0xbd9c : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      8      4      6      6      1      4      3      3      4      3      6   : 38048
aida02 :     11      8     10      3      4      4      3      4      3      3      6   : 37740
aida03 :      2      8      3      3      2      2      1      4      2      4      6   : 37720
aida04 :     26     15     17      4      4      2      1      4      2      4      6   : 38256
aida05 :     15      6      4      1      1      2      1      3      3      4      6   : 38156
aida06 :      7      7     16      6      3      4      2      2      3      4      6   : 38612
aida07 :     11      5     12      3      3      3      2      3      1      3      7   : 38836
aida08 :     13      6      7      2      1      2      2      2      3      4      6   : 37972
aida09 :     23      3      8      5      0      5      3      3      4      3      6   : 38164
aida10 :     12      5      7      1      2      2      4      3      2      4      6   : 37992
aida11 :     19      2      7      1      2      2      1      2      3      4      6   : 37740
aida12 :     21      5      3      3      3      3      1      2      3      4      6   : 37964
aida13 :     18      7      7      1      2      4      2      3      2      4      6   : 37776
aida14 :      8      3     13      7      2      4      2      3      2      4      6   : 37992
aida15 :     11      5     14      1      3      4      2      3      2      4      6   : 37908
aida16 :     17      6      6      2      2      3      1      2      3      4      6   : 37908

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19.30 Lost activity monitors - attachments 13-15

20.54 1.8.W spectra 20us FSR attachments 16 & 23
      1.8.W spectra 200us FSR attachments 17 & 22
      1.8.W spectra 2ms FSR attachments 18 & 21
      1.8.W spectra 20ms FSR attachments 19 & 20
Attachment 1: Screenshot_2021-05-22_AIDA_Alerting_-_Grafana(1).png
Screenshot_2021-05-22_AIDA_Alerting_-_Grafana(1).png
Attachment 2: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(11).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(11).png
Attachment 3: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(10).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(10).png
Attachment 4: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(9).png
Attachment 5: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(8).png
Attachment 6: Screenshot_2021-05-22_Statistics_aidas-gsi(9).png
Screenshot_2021-05-22_Statistics_aidas-gsi(9).png
Attachment 7: Screenshot_2021-05-22_Statistics_aidas-gsi(8).png
Screenshot_2021-05-22_Statistics_aidas-gsi(8).png
Attachment 8: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 9: 805.png
805.png
Attachment 10: 806.png
806.png
Attachment 11: R15_13
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  102550190 (   10129.0 Hz)
        Other data format:  159369822 (   15741.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      69714 (       6.9 Hz)
                              WR48-63:      69714 (       6.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:  159230394 (   15727.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:    10124.450 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000               67.951
  3                0.000                0.000
  4                0.000                0.000
  5                0.000             9971.491
  6                0.000                0.000
  7                0.000            10069.837
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0        213          0          0          0          0          0          0          0          0          0          0          0
  1       1159          2          0          0          0          0          1          1          0          0          0          0
  2        279   79660137          0          0          0          0      22470      22470          0   79615197          0          0
  3  101219504   79708933          0          0          0          0      46868      46868          0   79615197          0          0
  4        246          0          0          0          0          0          0          0          0          0          0          0
  5     588847        366          0          0          0          0        183        183          0          0          0          0
  6        280          0          0          0          0          0          0          0          0          0          0          0
  7     737827        382          0          0          0          0        191        191          0          0          0          0
  8        262          0          0          0          0          0          0          0          0          0          0          0
  9        221          0          0          0          0          0          0          0          0          0          0          0
 10        242          2          0          0          0          0          1          1          0          0          0          0
 11        210          0          0          0          0          0          0          0          0          0          0          0
 12        273          0          0          0          0          0          0          0          0          0          0          0
 13        193          0          0          0          0          0          0          0          0          0          0          0
 14        250          0          0          0          0          0          0          0          0          0          0          0
 15        184          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   45.816s (  698.440 blocks/s,  43.652 Mb/s)
Attachment 12: R15_19
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   55871232 (   17633.8 Hz)
        Other data format:  206048770 (   65032.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         56 (       0.0 Hz)
                               RESUME:         56 (       0.0 Hz)
                              SYNC100:      43803 (      13.8 Hz)
                              WR48-63:      43803 (      13.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:  205961052 (   65004.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:     3168.420 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000             2865.072
  2                0.000                0.000
  3                0.419                0.000
  4                0.000                0.000
  5                0.000             2166.184
  6                0.000                0.000
  7                0.000             1949.169
  8                0.000                0.000
  9                0.000                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0         55          0          0          0          0          0          0          0          0          0          0          0
  1     133932         56          0          0          0          0         28         28          0          0          0          0
  2         84  103025372          0          0          0          0      18982      18982          0  102987408          0          0
  3   42389344  103017790          0          0         56         56      22017      22017          0  102973644          0          0
  4         56          0          0          0          0          0          0          0          0          0          0          0
  5    6040914       2574          0          0          0          0       1287       1287          0          0          0          0
  6        103          0          0          0          0          0          0          0          0          0          0          0
  7    7306081       2976          0          0          0          0       1488       1488          0          0          0          0
  8         78          0          0          0          0          0          0          0          0          0          0          0
  9         86          0          0          0          0          0          0          0          0          0          0          0
 10         68          0          0          0          0          0          0          0          0          0          0          0
 11         75          0          0          0          0          0          0          0          0          0          0          0
 12        116          2          0          0          0          0          1          1          0          0          0          0
 13         80          0          0          0          0          0          0          0          0          0          0          0
 14         80          0          0          0          0          0          0          0          0          0          0          0
 15         80          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   43.406s (  737.221 blocks/s,  46.076 Mb/s)
Attachment 13: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(4).png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(4).png
Attachment 14: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(3).png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(3).png
Attachment 15: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(2).png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(2).png
Attachment 16: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(19).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(19).png
Attachment 17: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(18).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(18).png
Attachment 18: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(17).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(17).png
Attachment 19: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(16).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(16).png
Attachment 20: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(15).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(15).png
Attachment 21: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(14).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(14).png
Attachment 22: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(13).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(13).png
Attachment 23: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(12).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(12).png
  343   Fri May 21 23:19:23 2021 TDSaturday 22 May 00:00-08:00
00.16 413Gb remaining /TapeData filesystem
      estimate 413/2 x 177s = 10.15h to filesystem full - OK

00.18 All histograms, statistics and merger statistics zero'd

00.57 All system wide checks OK *except*

	 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x572 : 	 114  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19      9      3      4      1      5      2      2      3      3      6   : 36356
aida02 :      4      8      7      2      4      3      3      3      2      3      6   : 35968
aida03 :     12      9      6      2      5      3      2      3      3      3      6   : 36824
aida04 :     11      5      9      2      2      2      2      4      3      3      6   : 37028
aida05 :     24      9      2      1      2      3      2      3      4      3      6   : 37608
aida06 :     25      6      3      4      2      4      3      3      3      3      6   : 37060
aida07 :      3      3      3      1      3      3      2      2      2      2      7   : 37044
aida08 :     20      4      2      3      1      3      2      2      3      3      6   : 36016
aida09 :     15      8      2      5      0      3      3      2      3      3      6   : 36284
aida10 :     16      6      4      1      2      3      3      2      2      4      6   : 37328
aida11 :     20      2      2      1      1      3      2      2      3      3      6   : 35936
aida12 :      1      2      1      2      3      4      1      2      4      3      6   : 36900
aida13 :     25     10      4      1      0      3      2      3      3      3      6   : 36500
aida14 :     20      6      9      2      2      5      1      2      2      4      6   : 37200
aida15 :     31      9      5      2      3      3      1      3      3      3      6   : 36500
aida16 :     24      4      2      3      1      4      2      3      3      3      6   : 36672

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

01.12 Lost activity monitor - attachment 1
      1.8.H spectra - attachments 2 & 3
      1.8.L spectra - attachments 4-6
       aida01 pulser peak width 120 ch FWHM
      Rate specttra - attachments 7 & 8
01.06 Lost activity monitor - attachment 9
      ucesb - attachment 10
      Grafana - DSSSD bias & leakage currents past 7 days - attachment 11
      statistics - adc data, disc data, WR 28-47, pause, correlation scaler, good wave events - attachments 12-17
      FEE64 temperatures OK - attachment 18
      DSSSD bias & leakage currents OK - attachment 19
      Merger/Tape Server/Merger stats - attachment 20
       no merger warning/error messages since last restart
      
01.26 Analysis file S496/R14_1798 - attachment 21
       aida08 5.5% deadtime

01.17 Disk space remaining /TapeData filesystem

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  706M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.4T  395G  95% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 395/2 x 177s = 9.7h to filesystem full - OK

01.30 despec shift reports no data reported by ucesb
      aida stats & tape server ok - no data being forwarded to MBS by data relay

01.39 despec shift restart FATIMA DAQ - data forwarding to MBS by data relay resumes - AIDA file S496/R14_1811

04.35 System wide checks all OK *except*

Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x573 : 	 115  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  661M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.5T  337G  96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 337/2 x 177s = 8.3h to filesystem full - OK

FEE64 temperatures OK - attachment 22
ADC data item statistics OK - attachment 24
DSSSD bias & leakage current OK - attachment 26
analysis of file S496/R14_1920 - attachment 28
 aida04 deadtime 5.0%

07.10 Ssystem wide checks all OK *except*

	 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x573 : 	 115  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

[npg@aidas-gsi S496]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   18G   30G  37% /
tmpfs                 7.9G  662M  7.2G   9% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   90G  292G  24% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  6.5T  287G  96% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  5.6T  1.3T  82% /media/SecondDrive
/dev/sdc1             7.2T  6.6T  259G  97% /media/ThirdDrive

estimate 287/2 x 178s = 7.1h to filesystem full - OK

FEE64 temperatures OK - attachment 23
ADC data item statistics OK - attachment 25
DSSSD bias & leakage current OK - attachment 27
analysis of file S496/R14_1920 - attachment 29
 aida04 deadtime 4.4%

07.35 Found MBS data monitor kill'd
      Restarted MBS data relay and data monitor OK

08.00 beam off - end of S496

08.28 Tape Server -> no storage mode
      DAQ continues

09.58 All slow comparators -> 0x64
      BNC PB-5 OFF

      file S496/R15

      alpha background

      Tape server 200kb/s, merger 260k data items/s
Attachment 1: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(1).png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi(1).png
Attachment 2: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(7).png
Attachment 3: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(6).png
Attachment 4: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(5).png
Attachment 5: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi(1).png
Attachment 8: Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-22_Spectrum_Browser_aidas-gsi.png
Attachment 9: Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-22_Lost_activity_monitor_aidas-gsi.png
Attachment 10: Screenshot_2021-05-22_ucesb.png
Screenshot_2021-05-22_ucesb.png
Attachment 11: Screenshot_2021-05-22_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-22_AIDA_Alerting_-_Grafana.png
Attachment 12: Screenshot_2021-05-22_Statistics_aidas-gsi(5).png
Screenshot_2021-05-22_Statistics_aidas-gsi(5).png
Attachment 13: Screenshot_2021-05-22_Statistics_aidas-gsi(4).png
Screenshot_2021-05-22_Statistics_aidas-gsi(4).png
Attachment 14: Screenshot_2021-05-22_Statistics_aidas-gsi(3).png
Screenshot_2021-05-22_Statistics_aidas-gsi(3).png
Attachment 15: Screenshot_2021-05-22_Statistics_aidas-gsi(2).png
Screenshot_2021-05-22_Statistics_aidas-gsi(2).png
Attachment 16: Screenshot_2021-05-22_Statistics_aidas-gsi(1).png
Screenshot_2021-05-22_Statistics_aidas-gsi(1).png
Attachment 17: Screenshot_2021-05-22_Statistics_aidas-gsi.png
Screenshot_2021-05-22_Statistics_aidas-gsi.png
Attachment 18: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi.png
Attachment 19: 800.png
800.png
Attachment 20: 801.png
801.png
Attachment 21: R14_1798
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254662088 ( 1433466.5 Hz)
        Other data format:    7257912 (   40854.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        256 (       1.4 Hz)
                               RESUME:        257 (       1.4 Hz)
                              SYNC100:      32661 (     183.8 Hz)
                              WR48-63:      32661 (     183.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7192077 (   40483.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     365795 (    2059.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      177.655 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.112                0.000
  1                5.898                0.000
  2                0.060                0.000
  3                7.669                0.000
  4                0.073                7.902
  5                7.648                0.000
  6                0.356                0.000
  7                9.852                0.000
  8                1.513                0.000
  9                0.000               15.513
 10                2.639                0.000
 11                0.903                0.000
 12                0.687                0.000
 13                0.000                6.517
 14                0.073                0.000
 15                0.212                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5664289       2546          0          0          5          5        737        737          0       1062          0      19821
  1   24002581       6376          0          0         39         39       3149       3149          0          0          0      40581
  2   12140787    1878978          0          0          3          3       1758       1758          0    1875456          0      32794
  3   29094115    1803103          0          0         54         54       3917       3917          0    1795161          0      46229
  4    3034023    1008057          0          0          6          6        480        480          0    1007085          0       8526
  5   29014047     561174          0          0         54         54       3636       3636          0     553794          0      24714
  6   11191428    1180757          0          0          3          3       1528       1528          0    1177695          0      17521
  7   34103710     790567          0          0         51         52       4320       4320          0     781824          0      29424
  8   17182670       4348          0          0          9          9       2165       2165          0          0          0      41538
  9    2132311        520          0          0          0          0        260        260          0          0          0       3304
 10   25432181       6308          0          0         12         12       3142       3142          0          0          0      25046
 11   16063496       3832          0          0          8          8       1908       1908          0          0          0       7738
 12   11072472       2756          0          0          4          4       1374       1374          0          0          0      35884
 13    2789016        664          0          0          0          0        332        332          0          0          0       2683
 14   11165568       2934          0          0          4          4       1463       1463          0          0          0      24254
 15   20579394       4992          0          0          4          4       2492       2492          0          0          0       5738
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   41.474s (  771.565 blocks/s,  48.223 Mb/s)
Attachment 22: Screenshot_2021-05-22_Statistics_aidas-gsi(7).png
Screenshot_2021-05-22_Statistics_aidas-gsi(7).png
Attachment 23: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 24: Screenshot_2021-05-22_Statistics_aidas-gsi(6).png
Screenshot_2021-05-22_Statistics_aidas-gsi(6).png
Attachment 25: Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-22_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 26: 802.png
802.png
Attachment 27: 804.png
804.png
Attachment 28: R14_1867
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254778017 ( 1461534.6 Hz)
        Other data format:    7141983 (   40970.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        228 (       1.3 Hz)
                               RESUME:        228 (       1.3 Hz)
                              SYNC100:      32649 (     187.3 Hz)
                              WR48-63:      32649 (     187.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7076229 (   40592.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     358653 (    2057.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      174.322 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.023                0.000
  1                8.071                0.000
  2                0.956                0.000
  3                6.628                0.000
  4                0.186               12.353
  5                5.668                0.000
  6                0.255                0.000
  7                8.669                0.000
  8                1.570                0.000
  9                0.000               28.420
 10                2.125                0.000
 11                0.164                0.000
 12                0.103                0.000
 13                0.000               11.688
 14                0.094                0.000
 15                1.340                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5294176       2413          0          0          1          1        685        685          0       1041          0      18715
  1   22878067       6136          0          0         44         44       3024       3024          0          0          0      38998
  2   11959613    1840531          0          0          7          7       1681       1681          0    1837155          0      31909
  3   28914032    1781740          0          0         51         51       3845       3845          0    1773948          0      44969
  4    2970776     981532          0          0          7          7        454        454          0     980610          0       8409
  5   29112496     551278          0          0         40         40       3688       3688          0     543822          0      24330
  6   11438278    1149690          0          0          4          4       1643       1643          0    1146396          0      17361
  7   34470364     802023          0          0         41         41       4342       4342          0     793257          0      30254
  8   17110979       4312          0          0          6          6       2150       2150          0          0          0      40990
  9    2129568        524          0          0          0          0        262        262          0          0          0       3068
 10   25946722       6422          0          0          9          9       3202       3202          0          0          0      25071
 11   16268948       3960          0          0          3          3       1977       1977          0          0          0       7119
 12   11320269       2746          0          0          3          3       1370       1370          0          0          0      35684
 13    2799924        716          0          0          0          0        358        358          0          0          0       2469
 14   11389376       2772          0          0          2          2       1384       1384          0          0          0      23994
 15   20774429       5188          0          0         10         10       2584       2584          0          0          0       5313
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   43.000s (  744.186 blocks/s,  46.512 Mb/s)
Attachment 29: R14_1920
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254781729 ( 1428365.9 Hz)
        Other data format:    7138271 (   40018.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        226 (       1.3 Hz)
                               RESUME:        227 (       1.3 Hz)
                              SYNC100:      32665 (     183.1 Hz)
                              WR48-63:      32665 (     183.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7072488 (   39650.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     352885 (    1978.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      178.373 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.017                2.836
  1                6.610                0.000
  2                0.573                0.000
  3                7.826                0.000
  4                0.024               14.831
  5                7.215                0.000
  6                0.482                0.000
  7                6.876                0.000
  8                0.394                0.000
  9                0.000               17.832
 10                1.157                0.000
 11                1.395                0.000
 12                0.037                0.000
 13                0.004                9.016
 14                0.101                0.000
 15                1.578                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5306415       2397          0          0          1          1        665        665          0       1065          0      18998
  1   22746335       6008          0          0         36         36       2968       2968          0          0          0      37812
  2   11931383    1875609          0          0          5          5       1756       1756          0    1872087          0      31542
  3   28584441    1795513          0          0         57         57       3773       3773          0    1787853          0      43417
  4    2954259     971818          0          0          3          3        494        494          0     970824          0       8522
  5   28608195     543853          0          0         51         52       3567       3567          0     536616          0      24072
  6   11355091    1133078          0          0          6          6       1573       1573          0    1129920          0      16979
  7   34582740     783003          0          0         34         34       4406       4406          0     774123          0      29465
  8   17425265       4332          0          0          7          7       2159       2159          0          0          0      40811
  9    2130016        550          0          0          0          0        275        275          0          0          0       3176
 10   26903024       6616          0          0          4          4       3304       3304          0          0          0      24219
 11   16400506       4036          0          0          5          5       2013       2013          0          0          0       7163
 12   11085724       2934          0          0          1          1       1466       1466          0          0          0      35361
 13    2664218        650          0          0          1          1        324        324          0          0          0       2574
 14   11252748       2738          0          0          4          4       1365       1365          0          0          0      23421
 15   20851369       5136          0          0         11         11       2557       2557          0          0          0       5353
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   43.672s (  732.737 blocks/s,  45.796 Mb/s)
  342   Fri May 21 15:03:48 2021 CAFriday 21st May 16:00 - 00:00 shift
16:00 CA takes over
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

16:14 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x570 : 	 112  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

16:17 temperatures ok - attachment 1
      statistics ok - attachment 2
      detector bias / leakage currents ok - attachment 3
      analysis of R14_1625 - FEE7 deadtime at 4.5%

18:06 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x571 : 	 113  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:07 temperatures ok - attachment 4
      statistics ok - attachment 5
      detector bias / leakage currents ok - attachment 6
      analysis of R14_1662 - FEE7 deadtime at 4.3%

19:07 DAQ continues ok - writing to file R14_1682

20:07 system wide checks all ok *except*
      
      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x572 : 	 114 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

20:10 temperatures ok - attachment 7
      statistics ok - attachment 8
      detector bias / leakage currents ok - attachment 9
      analysis of R14_1703 - max deadtime at 4.0% - attachment 10

20:35 rate spectra - attachment 11
      HEC spectra - attachment 12 & 13

20:36 all histograms zeroed

20:37 DAQ continues ok - writing to file R14_1712

20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use.

      ~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files

21:12 no beam - writing to file R14_1723

21:17 beam is back

22:07 system wide checks all ok - WR decoder fault same as entry at 20:07

22:08 temperatures ok - attachment 14
      statistics ok - attachment 15
      detector bias / leakage currents ok - attachment 16
      analysis of R14_1740 - max deadtime at 4.2% - attachment 17








   
Attachment 1: Screenshot_from_2021-05-21_15-14-59.png
Screenshot_from_2021-05-21_15-14-59.png
Attachment 2: Screenshot_from_2021-05-21_15-15-38.png
Screenshot_from_2021-05-21_15-15-38.png
Attachment 3: Screenshot_from_2021-05-21_15-15-54.png
Screenshot_from_2021-05-21_15-15-54.png
Attachment 4: Screenshot_from_2021-05-21_17-06-56.png
Screenshot_from_2021-05-21_17-06-56.png
Attachment 5: Screenshot_from_2021-05-21_17-07-18.png
Screenshot_from_2021-05-21_17-07-18.png
Attachment 6: Screenshot_from_2021-05-21_17-07-47.png
Screenshot_from_2021-05-21_17-07-47.png
Attachment 7: Screenshot_from_2021-05-21_19-09-19.png
Screenshot_from_2021-05-21_19-09-19.png
Attachment 8: Screenshot_from_2021-05-21_19-09-59.png
Screenshot_from_2021-05-21_19-09-59.png
Attachment 9: Screenshot_from_2021-05-21_19-10-15.png
Screenshot_from_2021-05-21_19-10-15.png
Attachment 10: Screenshot_from_2021-05-21_19-14-45.png
Screenshot_from_2021-05-21_19-14-45.png
Attachment 11: Screenshot_from_2021-05-21_19-25-51.png
Screenshot_from_2021-05-21_19-25-51.png
Attachment 12: Screenshot_from_2021-05-21_19-30-14.png
Screenshot_from_2021-05-21_19-30-14.png
Attachment 13: Screenshot_from_2021-05-21_19-32-13.png
Screenshot_from_2021-05-21_19-32-13.png
Attachment 14: Screenshot_from_2021-05-21_21-08-28.png
Screenshot_from_2021-05-21_21-08-28.png
Attachment 15: Screenshot_from_2021-05-21_21-08-51.png
Screenshot_from_2021-05-21_21-08-51.png
Attachment 16: Screenshot_from_2021-05-21_21-09-06.png
Screenshot_from_2021-05-21_21-09-06.png
Attachment 17: Screenshot_from_2021-05-21_21-12-03.png
Screenshot_from_2021-05-21_21-12-03.png
  341   Fri May 21 07:03:50 2021 OH, TDFriday 21 May 08:00-16:00
08:00 OH and TD take over
      Current settings p+n sides at 0xc
      DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
      DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4

       PULSER SETTINGS
      ---------------------------
       Pulse is ON
       Positive Tail Pulse
       Trigger Source is Internal Clock
       Trigger Threshold is 3.5
       Amplitude   : 2.0 Volts
       Rep Rate    : 2.0 hZ
       Delay       : 250.0 ns
       Fall Time   : 1 ms
       Attenuation : 1
       Display is  : Volts
       Equivalent keV is : 200.0
       Ramp Start at 0.01 Volts
       Ramp Stop  at 9.99 Volts
       Ramp Start at 1.0 keV
       Ramp Stop  at 999.0 keV
       Ramp Time  is 60 seconds
       # Ramp Cycles is 1

08:29 System wide checks ok - WR difference at 111
      Statistics ok (Still high following the sharp rise at 18:30 yesterday) - attachment 1
      Temperature ok - attachment 2
      Bias and leakage current ok - attachment 3

09:58 System wide checks ok - WR difference at 111
      Statistics ok - attachment 4
      Temperature ok - attachment 5
      Bias and leakage current ok - attachment 6

      Max deadtime this morning has been around 5.5% on FEE8

12:32 System wide checks ok - WR difference at 111
      Statistics ok - attachment 7
      Temperature ok - attachment 9
      Bias and leakage current ok - attachment 8
Attachment 1: 210521_0829_Stats.png
210521_0829_Stats.png
Attachment 2: 210521_0829_Temp.png
210521_0829_Temp.png
Attachment 3: 210521_0830_Bias.png
210521_0830_Bias.png
Attachment 4: 210521_0957.png
210521_0957.png
Attachment 5: 210521_0957_Temp.png
210521_0957_Temp.png
Attachment 6: 210521_0958_Bias.png
210521_0958_Bias.png
Attachment 7: 210521_1232_Stats.png
210521_1232_Stats.png
Attachment 8: 210521_1233_Bias.png
210521_1233_Bias.png
Attachment 9: 210521_1233_Temp.png
210521_1233_Temp.png
  340   Fri May 21 06:09:28 2021 MLsystem wide checks

System  wide checks at 7am CET

All seems fine, nothing significant to report

Slight increase of dead time noted in FEE 3, 5 and 7.

-----
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

-----

ADC calibration

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


-----

White Rabbit:

         Base         Current     Difference
aida05 fault      0x500 :      0x56e :      110  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-----

FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-----

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     23      5      6      5      0      4      1      3      3      3      6   : 36484
aida02 :     10      3      3      2      3      2      2      2      3      3      6   : 35952
aida03 :      2      5      1      4      5      2      2      3      3      3      6   : 36608
aida04 :      1      3      4      0      4      3      2      3      3      3      6   : 36572
aida05 :     19      9      3      2      2      3      1      3      4      3      6   : 37380
aida06 :     13     11      1      5      3      4      4      2      3      3      6   : 36860
aida07 :     12      2      4      3      3      2      1      3      2      2      7   : 37280
aida08 :     25      5      7      1      1      3      3      3      3      3      6   : 36828
aida09 :     22      5      6      4      1      3      2      3      3      3      6   : 36640
aida10 :     23      8      6      3      2      3      2      2      2      4      6   : 37212
aida11 :     24     14      7      3      3      2      3      3      3      3      6   : 36960
aida12 :     16     10      5      3      1      2      1      2      4      3      6   : 36736
aida13 :     23      6      4      2      2      3      2      3      3      3      6   : 36620
aida14 :     22      7      8      4      3      4      1      2      2      4      6   : 37200
aida15 :     10     12      9      2      1      3      1      3      3      3      6   : 36376
aida16 :     18      5      6      2      0      3      2      4      3      3      6   : 37008

 

Attachment 1: 2021-05-21_7h00_LeakCurrent.png
2021-05-21_7h00_LeakCurrent.png
Attachment 2: 2021-05-21_7h00_Stats_GoodEvents.png
2021-05-21_7h00_Stats_GoodEvents.png
Attachment 3: 2021-05-21_7h00_Stats_ADCitems.png
2021-05-21_7h00_Stats_ADCitems.png
Attachment 4: 2021-05-21_7h00_stats_Correlation.png
2021-05-21_7h00_stats_Correlation.png
Attachment 5: 2021-05-21_7h00_Stats_PAUSE.png
2021-05-21_7h00_Stats_PAUSE.png
Attachment 6: 2021-05-21_7h00_Stats_RESUME.png
2021-05-21_7h00_Stats_RESUME.png
Attachment 7: 2021-05-21_7h00_temperature.png
2021-05-21_7h00_temperature.png
Attachment 8: 2021-05-21_7h00_ucesb.png
2021-05-21_7h00_ucesb.png
  339   Fri May 21 02:17:47 2021 MLsystem wide checks

Wide check at 3am -

All fine.

Same output as at the beginning of the shift (see my previous entry) and attachments.

 

Attachment 1: 2021-05-21_3h05_Leakcurrent.png
2021-05-21_3h05_Leakcurrent.png
Attachment 2: 2021-05-21_3h05_stats_adcItem.png
2021-05-21_3h05_stats_adcItem.png
Attachment 3: 2021-05-21_3h05_stats_Correlation.png
2021-05-21_3h05_stats_Correlation.png
Attachment 4: 2021-05-21_3h05_stats_GoodEvents.png
2021-05-21_3h05_stats_GoodEvents.png
Attachment 5: 2021-05-21_3h05_stats_PAUSE.png
2021-05-21_3h05_stats_PAUSE.png
Attachment 6: 2021-05-21_3h05_stats_RESUME.png
2021-05-21_3h05_stats_RESUME.png
Attachment 7: 2021-05-21_3h05_temperature.png
2021-05-21_3h05_temperature.png
Attachment 8: 2021-05-21_3h05_ucesb.png
2021-05-21_3h05_ucesb.png
Attachment 9: 2021-05-21_3h05_aidaGrafana.png
2021-05-21_3h05_aidaGrafana.png
  338   Thu May 20 23:26:11 2021 Marc Labiche (ML)system wide checks

System Wide check at begining of the night shift:

-----

Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

-----

ADC calibration:
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

-----

White rabbit status:

         Base         Current     Difference
aida05 fault      0x500 :      0x56d :      109  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-----

FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-----

Memory info from FEE cards

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      7      7      6      1      5      2      2      3      3      6   : 36472
aida02 :     19      5      4      4      4      3      3      3      2      3      6   : 36020
aida03 :     22     10      2      2      5      2      2      3      3      3      6   : 36680
aida04 :     12      7      1      0      4      3      1      4      3      3      6   : 36856
aida05 :     16      7      5      3      1      3      2      3      3      3      6   : 36584
aida06 :     20      7      5      6      2      4      4      2      3      3      6   : 36888
aida07 :     13      7      5      2      3      2      1      3      2      2      7   : 37308
aida08 :     18      7      5      3      0      2      3      3      3      3      6   : 36656
aida09 :     21      6      4      4      1      4      2      3      3      3      6   : 36740
aida10 :     21     10      2      3      2      3      2      2      2      4      6   : 37156
aida11 :     23      2      4      4      3      2      2      3      2      4      6   : 37612
aida12 :     24      1      4      3      2      2      1      2      4      3      6   : 36744
aida13 :     26      9      4      2      1      4      3      2      3      3      6   : 36464
aida14 :     19      9      9      2      3      4      1      2      2      4      6   : 37156
aida15 :     10      7     10      0      1      3      1      3      3      3      6   : 36288
aida16 :     28      7      3      2      1      3      3      3      3      3      6   : 36824

Attachment 1: 2021-05-21_0h20_ucesb.png
2021-05-21_0h20_ucesb.png
Attachment 2: 2021-05-21_0h10_Leakcurrent.png
2021-05-21_0h10_Leakcurrent.png
Attachment 3: 2021-05-21_0h10_Stat_adcItem.png
2021-05-21_0h10_Stat_adcItem.png
Attachment 4: 2021-05-21_0h10_Stats_Correlation.png
2021-05-21_0h10_Stats_Correlation.png
Attachment 5: 2021-05-21_0h10_Stats_GoodEvents.png
2021-05-21_0h10_Stats_GoodEvents.png
Attachment 6: 2021-05-21_0h10_Stats_PAUSE.png
2021-05-21_0h10_Stats_PAUSE.png
Attachment 7: 2021-05-21_0h10_stats_RESUME.png
2021-05-21_0h10_stats_RESUME.png
Attachment 8: 2021-05-21_0h10_temperature.png
2021-05-21_0h10_temperature.png
Attachment 9: 2021-05-21_0h30_AIDAgrafana.png
2021-05-21_0h30_AIDAgrafana.png
  337   Thu May 20 14:56:33 2021 OHThursday 20th May 16:00 - 24:00
16:00 p+n threshold 0xc (120keV)
      n+n threshold DSSD2 0x20 (250keV) ASICS 1-3, 0x64 ASIC4
      n+n threshold DSSD2 0x19 (250keV) ASICS 1-3, 0x64 ASIC4

      Dead time has increased from yesterday. From the elog this appears to coincide with their changing of the spill structure.

17:30 Grafana informs us we have hit 10uA on DSSD2. This is fine with the size of the detector

18:12 Gregor noticed that at 18:02 the rate in AIDA increased. This could be related to the autofill starting around now.
      The deadtimes in FEE6 and 8 have risen to around 6.25%.
      Looking at this mornings and yesterday evenings fill times this behavior isn't observed

18:30 The rates still seem to be increased so is unlikely to be due to the filling which finished at least 10 minutes ago

18:49 Have tried raising the thresholds to 0x1a for the y side strips of DSSD2
      This has brought the deadtime of AIDA06 down to 5.53% (This seems to have just been a statistical fluctuation and the rate went back up again in the next file)

18:12 Raised threshold to 0x1b for ASIC1-3 of FEE6 and 8

18:15 System wide checks ok except WR as usual
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56d : 	 109  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics increased at 18:02 still working on - attachment 2
      FEE temp - attachment 3
      Bias and leakage current - attachment 4


20:51 System wide checks ok - WR same as before
      Statistics still high - attachment 5
      Temperatures ok - attachment 6
      Bias and leakage currents ok - attachment 7 

22:52 Statistics still high - attachment 8
      Temp ok - attachment 9
      Bias and leakage current ok - attachment 10
      System wide checks ok - WR difference still 109
Attachment 1: 210520_18002_MBSRate.png
210520_18002_MBSRate.png
Attachment 2: 210520_1916_Stats.png
210520_1916_Stats.png
Attachment 3: 210520_1916_Temp.png
210520_1916_Temp.png
Attachment 4: 210520_1917_Bias.png
210520_1917_Bias.png
Attachment 5: 21050_2052_Stats.png
21050_2052_Stats.png
Attachment 6: 210520_2052_Temp.png
210520_2052_Temp.png
Attachment 7: 210520_2053_Bias.png
210520_2053_Bias.png
Attachment 8: 210520_2251_Stats.png
210520_2251_Stats.png
Attachment 9: 210520_2251_Temp.png
210520_2251_Temp.png
Attachment 10: 210520_2254_Bias.png
210520_2254_Bias.png
  336   Thu May 20 07:03:02 2021 CAThursday May 20th 08:00 - 16:00 shift
08:00 CA takes over

08:04 DAQ continues ok - writing to file R14_989

09:30 all system wide checks ok *except* WR decoder status;

      
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56a : 	 106  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

09:34 temperatures ok - attachment 1
      stats ok - attachment 2
      bias/leakage currents ok - attachment 3
      R14_1016 analysis - max deadtime ~4.3% - attachment 4

11:30 all system wide checks ok *except* WR decoder status;

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56b : 	 107  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

11:34 temperatures ok - attachment 5
      stats ok - attachment 6
      bias/leakage currents ok - attachment 7
      R14_1052 analysis - max deadtime ~3.6% - attachment 8

13:05 beam optimisation currently taking place
      DAQ continues ok and writing to file R14_1085

13:17 no beam

13:28 all system wide checks ok, WR decoder status difference same as entry at 11:30.

13:31 temperatures ok - attachment 9
      stats ok - attachment 10
      bias/leakage currents ok - attachment 11
      R14_1085 analysis (beam off) - max deadtime ~1.34% - attachment 12

13:37 beam is back - rate spectra - attachment 13
      DAQ ok - writing to file R14_1088

13:43 analysis of R14_1088 - max deadtime back up to ~4.5% - attachment 14

14:21 beam off - issue with UNILAC
      DESPEC also report issues with their DAQ
      AIDA continues to forward data to MBS ok - lower rates while beam off

14.30 1.8.H spectra - attachments 16-18
      1.8.L spectra - attachments 19-22
       aida01 pulser peak width 99 ch FWHM
      Rate spectra - attachment 23
      Grafana - DSSSD bias & leakage currents past 7 days - attachment 24
      DSSSD bias & leakage currents - attachment 25
      Merger/Tape Server/Merger stats - attachment 26



14.41 All histograms & stats zero'd

14.55 Lost Activity Monitors - attachments 27, 28 & 35
      Stats - ADC data items, disc, good wave, WR 28-47, pause, correlation scaler

15:00 beam is back

15:05 AIDA writing to file R14_1108
      ADC data items - attachment 36

15:27 system wide checks ok *except* WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x56c : 	 108  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

15:30 temperatures ok - attachment 37
      stats ok - attachment 38
      bias/leakage currents ok - attachment 39
      R14_1113 analysis (beam off) - max deadtime ~4% - attachment 40
Attachment 1: Screenshot_from_2021-05-20_08-33-49.png
Screenshot_from_2021-05-20_08-33-49.png
Attachment 2: Screenshot_from_2021-05-20_08-34-10.png
Screenshot_from_2021-05-20_08-34-10.png
Attachment 3: Screenshot_from_2021-05-20_08-34-44.png
Screenshot_from_2021-05-20_08-34-44.png
Attachment 4: Screenshot_from_2021-05-20_08-37-13.png
Screenshot_from_2021-05-20_08-37-13.png
Attachment 5: Screenshot_from_2021-05-20_10-32-22.png
Screenshot_from_2021-05-20_10-32-22.png
Attachment 6: Screenshot_from_2021-05-20_10-32-47.png
Screenshot_from_2021-05-20_10-32-47.png
Attachment 7: Screenshot_from_2021-05-20_10-33-04.png
Screenshot_from_2021-05-20_10-33-04.png
Attachment 8: Screenshot_from_2021-05-20_10-36-31.png
Screenshot_from_2021-05-20_10-36-31.png
Attachment 9: Screenshot_from_2021-05-20_12-29-26.png
Screenshot_from_2021-05-20_12-29-26.png
Attachment 10: Screenshot_from_2021-05-20_12-29-49.png
Screenshot_from_2021-05-20_12-29-49.png
Attachment 11: Screenshot_from_2021-05-20_12-30-20.png
Screenshot_from_2021-05-20_12-30-20.png
Attachment 12: Screenshot_from_2021-05-20_12-33-48.png
Screenshot_from_2021-05-20_12-33-48.png
Attachment 13: Screenshot_from_2021-05-20_12-38-42.png
Screenshot_from_2021-05-20_12-38-42.png
Attachment 14: Screenshot_from_2021-05-20_12-38-42.png
Screenshot_from_2021-05-20_12-38-42.png
Attachment 15: Screenshot_from_2021-05-20_12-43-18.png
Screenshot_from_2021-05-20_12-43-18.png
Attachment 16: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(7).png
Attachment 17: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(6).png
Attachment 18: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(5).png
Attachment 19: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(4).png
Attachment 20: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(3).png
Attachment 21: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(2).png
Attachment 22: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi(1).png
Attachment 23: Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-20_Spectrum_Browser_aidas-gsi.png
Attachment 24: Screenshot_2021-05-20_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-20_AIDA_Alerting_-_Grafana.png
Attachment 25: 710.png
710.png
Attachment 26: 711.png
711.png
Attachment 27: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(2).png
Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(2).png
Attachment 28: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(1).png
Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi(1).png
Attachment 29: Screenshot_2021-05-20_Statistics_aidas-gsi(5).png
Screenshot_2021-05-20_Statistics_aidas-gsi(5).png
Attachment 30: Screenshot_2021-05-20_Statistics_aidas-gsi(4).png
Screenshot_2021-05-20_Statistics_aidas-gsi(4).png
Attachment 31: Screenshot_2021-05-20_Statistics_aidas-gsi(3).png
Screenshot_2021-05-20_Statistics_aidas-gsi(3).png
Attachment 32: Screenshot_2021-05-20_Statistics_aidas-gsi(2).png
Screenshot_2021-05-20_Statistics_aidas-gsi(2).png
Attachment 33: Screenshot_2021-05-20_Statistics_aidas-gsi(1).png
Screenshot_2021-05-20_Statistics_aidas-gsi(1).png
Attachment 34: Screenshot_2021-05-20_Statistics_aidas-gsi.png
Screenshot_2021-05-20_Statistics_aidas-gsi.png
Attachment 35: Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi.png
Screenshot_2021-05-20_Lost_activity_monitor_aidas-gsi.png
Attachment 36: Screenshot_from_2021-05-20_14-07-22.png
Screenshot_from_2021-05-20_14-07-22.png
Attachment 37: Screenshot_from_2021-05-20_14-28-33.png
Screenshot_from_2021-05-20_14-28-33.png
Attachment 38: Screenshot_from_2021-05-20_14-29-04.png
Screenshot_from_2021-05-20_14-29-04.png
Attachment 39: Screenshot_from_2021-05-20_14-29-18.png
Screenshot_from_2021-05-20_14-29-18.png
Attachment 40: Screenshot_from_2021-05-20_14-31-50.png
Screenshot_from_2021-05-20_14-31-50.png
  335   Thu May 20 01:00:30 2021 MuneerahThu May 20 00:00-08:00

02:00 System check

The ucesb has stopped around 00:26, the team in the lab ask us to restart AIDA! Called OH for this but the team in the lab figured out thats no need to restart AIDA!

The rates on ucesb got back to work by around 01:12

Deat Time check ok R14_870, attachment 1

Statistics are ok attachment 2

Temperatures are ok attacment 3

Voltages are ok attachment 4

 

system wide check

All ok except

ADC calibration all failed

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White rabbite

1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      8      8      5      0      6      1      2      2      2      7   : 37296
aida02 :     19      8     11      2      3      4      2      2      3      3      6   : 36412
aida03 :     14      6      3      4      1      3      1      3      3      3      6   : 36312
aida04 :     25     12     12      4      3      4      1      4      3      3      6   : 37316
aida05 :      4      8      3      4      1      2      2      4      3      3      6   : 36928
aida06 :     42     12      8      7      2      3      3      4      3      3      6   : 37736
aida07 :     24      6      7      3      3      3      1      3      2      2      7   : 37536
aida08 :     10     15      7      2      1      4      2      3      3      3      6   : 36752
aida09 :     29     13      6      6      4      3      3      3      3      3      6   : 37244
aida10 :     18     12      7      2      2      3      3      2      2      4      6   : 37464
aida11 :     20      8      0      5      5      4      2      3      3      3      6   : 36976
aida12 :      7      2      4      3      2      3      2      2      4      3      6   : 37068
aida13 :      2      5      5      4      2      3      2      3      3      3      6   : 36608
aida14 :      9     13      8      2      2      2      3      3      2      4      6   : 37836
aida15 :     24     12     12      1      0      3      2      3      3      3      6   : 36640
aida16 :     12      9      9      2      1      3      2      4      3      3      6   : 37128

6:45 System check

Temperatures, Voltages, Histograms, Statistics, dead time are all ok attachments 5, 6, 7, 8, 9

System wide check are all ok except:

ADC calibration all failed

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit 1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20     11      3      5      0      6      1      2      3      2      7   : 38264
aida02 :     14      9      9      2      3      3      3      2      3      3      6   : 36496
aida03 :     13      7      3      5      2      3      2      3      3      3      6   : 36668
aida04 :     30     15     13      4      3      4      1      3      4      3      6   : 37888
aida05 :     17      9      3      3      1      2      2      4      3      3      6   : 36956
aida06 :     40     16      5      6      2      3      2      3      2      4      6   : 37936
aida07 :      3      2      3      2      3      4      2      2      2      2      7   : 37196
aida08 :     17     10      6      2      1      3      3      3      3      3      6   : 36852
aida09 :     29     11      7      6      3      3      2      4      3      3      6   : 37436
aida10 :      1     10      5      1      1      3      3      2      2      4      6   : 37252
aida11 :     30      9      0      5      4      4      2      3      3      3      6   : 36960
aida12 :     19     10      5      3      2      3      2      2      4      3      6   : 37196
aida13 :     22      9      6      3      2      3      3      3      3      3      6   : 36960
aida14 :     19      6      9      2      2      3      3      2      2      4      6   : 37452
aida15 :      6      8     11      2      1      3      2      3      3      3      6   : 36616
aida16 :     19     14      5      3      1      3      1      4      3      3      6   : 36908

 

07:33 system check

Deat time, voltages, statistics, temeratuers are all ok, attachment 10, 11, 12, 13

System wide check

all ok except

ADC all failed

White rabbit

1 failed

   
         Base         Current     Difference
aida05 fault      0x500 :      0x569 :      105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      9      4      6      0      5      1      2      2      2      7   : 37152
aida02 :     19      6     13      1      3      3      4      2      3      3      6   : 36780
aida03 :     23      5      6      4      2      3      1      3      3      3      6   : 36452
aida04 :     24     11     14      3      3      4      2      3      3      3      6   : 37048
aida05 :     17      3      6      3      1      2      2      3      3      3      6   : 36444
aida06 :     36     15      4      9      2      4      2      4      3      3      6   : 37608
aida07 :     20      8      5      3      3      4      2      2      2      2      7   : 37376
aida08 :     21     11     10      2      1      4      2      2      2      4      6   : 37324
aida09 :     27     14      7      6      3      3      2      4      3      3      6   : 37452
aida10 :     25      9      5      2      2      3      3      2      2      4      6   : 37436
aida11 :     30     14      2      4      5      3      2      3      3      3      6   : 36936
aida12 :     22      3      5      3      2      3      2      3      3      3      6   : 36640
aida13 :     10      7      1      4      2      3      3      2      3      3      6   : 36336
aida14 :     13      9     12      2      3      3      2      3      2      4      6   : 37820
aida15 :     20      8     11      3      0      3      2      3      3      3      6   : 36640
aida16 :     23      6      6      2      0      2      1      3      4      3      6   : 37164

 

 

 

 

 

Attachment 1: 210520_0156_DT.png
210520_0156_DT.png
Attachment 2: 210520_0157_Stats.png
210520_0157_Stats.png
Attachment 3: 210520_0157_Temps.png
210520_0157_Temps.png
Attachment 4: 210520_0157_Voltages.png
210520_0157_Voltages.png
Attachment 5: 210520_0537_Temps.png
210520_0537_Temps.png
Attachment 6: 210520_0539_VOltages.png
210520_0539_VOltages.png
Attachment 7: 210520_0540_Histo.png
210520_0540_Histo.png
Attachment 8: 210520_0541_Rates.png
210520_0541_Rates.png
Attachment 9: 210520_0542_DT.png
210520_0542_DT.png
Attachment 10: 210520_0730_DT.png
210520_0730_DT.png
Attachment 11: 210520_0730_Voltages.png
210520_0730_Voltages.png
Attachment 12: 210520_0731_Stats.png
210520_0731_Stats.png
Attachment 13: 210520_0731_Tems.png
210520_0731_Tems.png
  334   Wed May 19 20:02:27 2021 NHAIDA Dead Time Analysis
Some plots of AIDA dead time (From PAUSE/RESUME data items) against time.

The histograms are binned in intervals of 100 ms, with the blue bars corresponding to dead time in the FEE - the y-value is the percent of the 100ms interval the card was dead.

Beam spills are indicated by the red bars (1 = on spill)

Dead time is mostly dominant in the n+n FEES: fee2, fee4, fee6, fee8 and a 30 second snapshot is shown
Can see deadtime largely seems to occur at the end of a spill - this suggests buffers/queues filling up from the high on-spill rates .

The much quieter p+n strips show a similar pattern but much less frequently.

Deadtime does not always sync up between FEE64s it seems.

IN all case the pause/resume interval seems to only last for less than 100 ms.
Attachment 1: dt_fee2.png
dt_fee2.png
Attachment 2: dt_fee4.png
dt_fee4.png
Attachment 3: dt_fee6.png
dt_fee6.png
Attachment 4: dt_fee8.png
dt_fee8.png
Attachment 5: dt_fee9.png
dt_fee9.png
Attachment 6: dt_fee9_lr.png
dt_fee9_lr.png
  333   Wed May 19 18:50:01 2021 LSWednesday 19th May 20:00 to 24:00
20.00
Stats okay (Fig 1)
Analysed R14_762 deadtimes are okay (Fig 2)

20.30
Stats okay (Fig 3)
Analysed R14_771 deadtimes are okay (Fig 4)

21.00
Stats okay (Fig 5)
FEE temps okay (Fig 6)
Bias (Fig 7)
Rate spectra (Fig 8)

System wide checks okay except white rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x569 : 	 105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~2.8M items/s
Tape server ~10MB/s

Analysed R14_781 deadtimes are okay (Fig 9)

21.30
Stats okay (Fig 10)
Analysed R14_789 deadtimes are okay (Fig 11)

22.00
Stats okay (Fig 12)
Analysed R14_799 deadtimes are okay (Fig 13)

22.30
Stats okay (Fig 14)
Analysed R14_807 deadtimes are okay (Fig 15)

23.00
Stats okay (Fig 16)
FEE temps okay (Fig 17)
Bias (Fig 18)

System wide checks okay except white rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x569 : 	 105  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~3M items/s
Tape server ~10MB/s

Analysed R14_819 deadtimes are okay (Fig 19)

23.30
Stats okay (Fig 20)
Analysed R14_827 deadtimes are okay (Fig 21)
Attachment 1: 210519_Stats_2000.png
210519_Stats_2000.png
Attachment 2: Deadtimes_R14_762.png
Deadtimes_R14_762.png
Attachment 3: 210519_Stats_2030.png
210519_Stats_2030.png
Attachment 4: Deadtimes_R14_771.png
Deadtimes_R14_771.png
Attachment 5: 210519_Stats_2100.png
210519_Stats_2100.png
Attachment 6: 210519_FEEtemps_2100.png
210519_FEEtemps_2100.png
Attachment 7: 210519_BIAS_2100.png
210519_BIAS_2100.png
Attachment 8: 210519_Spectra_2100.png
210519_Spectra_2100.png
Attachment 9: deadtimes_R14_781.png
deadtimes_R14_781.png
Attachment 10: 210519_Stats_2130.png
210519_Stats_2130.png
Attachment 11: deadtimes_R14_789.png
deadtimes_R14_789.png
Attachment 12: 210519_Stats_2200.png
210519_Stats_2200.png
Attachment 13: deadtimes_R14_799.png
deadtimes_R14_799.png
Attachment 14: 210519_Stats_2230.png
210519_Stats_2230.png
Attachment 15: deatimes_R14_807.png
deatimes_R14_807.png
Attachment 16: 210519_Stats_2300.png
210519_Stats_2300.png
Attachment 17: 210519_FEEtemps_2300.png
210519_FEEtemps_2300.png
Attachment 18: 210519_bias_2300.png
210519_bias_2300.png
Attachment 19: deatimes_R14_819.png
deatimes_R14_819.png
Attachment 20: 210519_Stats_2330.png
210519_Stats_2330.png
Attachment 21: deadtimes_R14_827.png
deadtimes_R14_827.png
  332   Wed May 19 14:54:36 2021 CBWednesday 19 May 16:00 - 20:00
15:55 Took over from OH

16:00
Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System wide checks
All OK, except ADC all fail (as before) and WR

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x567 : 	 103  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_691. Attach 4. Dead time <5%

16:34 MBS changes file.


17:00
Stats OK - attach 5
Temps OK - as before
System wide checks as before
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


17:55
Stats OK - attach 6
Temps OK - as before

Leakage current approaching 10 uA - attach 7

System-wide checks - as before except
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x568 : 	 104  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



18:01 MBS switches to new file



18:52
Stats OK - attach 8
Temps OK - as before

System-wide checks as before

Analysed R14_741. Avg. dead time slightly increased to 5.1%. Attach 9

19:36 Stats OK - attach 10
Bias - attach 11

19:48 Shift handed over to LS
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Temps1.png
Temps1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Analysis_R14_691
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  251761346 (  995725.1 Hz)
        Other data format:   10158654 (   40177.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        199 (       0.8 Hz)
                               RESUME:        199 (       0.8 Hz)
                              SYNC100:      32942 (     130.3 Hz)
                              WR48-63:      32942 (     130.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   10092372 (   39915.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     503210 (    1990.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      252.842 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.053               17.526
  1                1.803                0.000
  2                0.604                2.918
  3                3.773                0.000
  4                0.070               17.512
  5                9.601                0.000
  6                0.278                0.000
  7                9.149                0.000
  8                0.114                0.000
  9                0.000               64.964
 10                0.536                0.000
 11                0.070                0.000
 12                0.024                6.415
 13                0.000               66.391
 14                0.036                7.395
 15                0.097                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6023981       3019          0          0          3          3        749        749          0       1515          0      26768
  1   22620294       5924          0          0         14         14       2948       2948          0          0          0      55394
  2   11082857    2607083          0          0          5          5       1799       1799          0    2603475          0      43681
  3   30564809    2580290          0          0         26         26       4295       4295          0    2571648          0      65104
  4    3958862    1393424          0          0          7          7        678        678          0    1392054          0      11940
  5   37388332     780379          0          0         62         62       4752       4752          0     770751          0      33878
  6   10191568    1631763          0          0          8          8       1480       1480          0    1628787          0      23955
  7   39960208    1134610          0          0         55         55       5179       5179          0    1124142          0      42303
  8   17144158       4230          0          0          3          3       2112       2112          0          0          0      56568
  9    2458065        662          0          0          0          0        331        331          0          0          0       4321
 10   20331742       4810          0          0          6          6       2399       2399          0          0          0      34970
 11   11809904       2968          0          0          4          4       1480       1480          0          0          0      10406
 12    9134013       2250          0          0          1          1       1124       1124          0          0          0      49240
 13    2508032        620          0          0          0          0        310        310          0          0          0       3587
 14    8541585       2182          0          0          2          2       1089       1089          0          0          0      33394
 15   18042936       4440          0          0          3          3       2217       2217          0          0          0       7701
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.672s (  950.348 blocks/s,  59.397 Mb/s)
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Stats3.png
Stats3.png
Attachment 7: Bias3.png
Bias3.png
Attachment 8: Analysis_R14_741
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  253670855 ( 1280798.3 Hz)
        Other data format:    8249145 (   41650.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        236 (       1.2 Hz)
                               RESUME:        236 (       1.2 Hz)
                              SYNC100:      32737 (     165.3 Hz)
                              WR48-63:      32737 (     165.3 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8183199 (   41317.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     413031 (    2085.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      198.057 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.011                2.771
  1                4.649                0.000
  2                0.028                0.000
  3                4.808                0.000
  4                0.111                7.488
  5                7.093                0.000
  6                0.128                0.000
  7               10.340                0.000
  8                0.971                0.000
  9                0.000               24.260
 10                0.713                0.000
 11                0.245                0.000
 12                0.122                0.000
 13                0.000               21.524
 14                0.232                0.000
 15                1.605                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5709345       2725          0          0          1          1        769        769          0       1185          0      22625
  1   23799538       6034          0          0         27         27       2990       2990          0          0          0      44162
  2   11722022    2092139          0          0          1          1       1767       1767          0    2088603          0      36240
  3   29654318    2044332          0          0         38         38       3940       3940          0    2036376          0      53125
  4    3438629    1158241          0          0          8          8        579        579          0    1157067          0       9655
  5   33876189     636041          0          0         60         60       4396       4396          0     627129          0      27488
  6   10663472    1364583          0          0          5          5       1426       1426          0    1361721          0      20341
  7   38937063     921272          0          0         54         54       5023       5023          0     911118          0      34159
  8   16044621       3966          0          0          9          9       1974       1974          0          0          0      46740
  9    2236927        610          0          0          0          0        305        305          0          0          0       3583
 10   23087850       5792          0          0          6          6       2890       2890          0          0          0      28946
 11   13996775       3362          0          0          8          8       1673       1673          0          0          0       8557
 12    9568717       2316          0          0          4          4       1154       1154          0          0          0      40620
 13    2488330        626          0          0          0          0        313        313          0          0          0       2902
 14    9604847       2392          0          0          6          6       1190       1190          0          0          0      27522
 15   18842212       4714          0          0          9          9       2348       2348          0          0          0       6366
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.094s (  938.588 blocks/s,  58.662 Mb/s)
Attachment 9: Stats4.png
Stats4.png
Attachment 10: Stats6.png
Stats6.png
Attachment 11: Bias6.png
Bias6.png
  331   Wed May 19 11:24:55 2021 RDPMonday 19 April 12 noon shift
System checks gave 11 passed/1 failed, except for ADC timimgs: 10 passed/2 failed.

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2


		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd056 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x16 : 	 22  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      2      0      5      5      1      2      3      3      6   : 36120
aida02 :     25     10      0      4      3      5      2      3      3      3      6   : 36980
aida03 :     22     12      5      2      1      3      2      3      3      3      6   : 36616
aida04 :     16      3      3      3      4      4      4      2      3      3      6   : 36840
aida05 :     14      5      6      2      3      4      4      2      3      3      6   : 36800
aida06 :     30     10      6      3      3      4      2      3      3      3      6   : 36936
aida07 :     27     11      1      0      0      3      2      3      3      3      6   : 36436
aida08 :     38      7      7      3      3      2      2      3      3      3      6   : 36704
aida09 :      4      6      4      2      4      4      1      4      3      3      6   : 37056
aida10 :     21     10      3      3      1      5      1      3      3      3      6   : 36596
aida11 :     24      8      2      1      4      3      2      4      2      3      6   : 36192
aida12 :     31     12      4      3      5      5      2      2      3      3      6   : 36668

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Mon Apr 19 10:32:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

There should be a separate elog note about changing one of the bias voltages.

All histograms zeroed at 16:13.

16.20 DAQ continues file NULL/R33_156
      aida09 HEC fast comparator 0x2->0x1
      DSSSD#3 bias -100->-120V
Attachment 1: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 2: Screenshot_2021-04-19_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2021-04-19_Tape_Service_(Expert)_aidas-gsi.png
Attachment 3: 62.png
62.png
  330   Wed May 19 09:12:52 2021 Muneerah, Betool, LS17th May 00:00-12:00
Auther: Muneerah and Betool
02:00 The dead time have been checked every 30 min from the start of the shift, it seemes Ok
      Statistics ok - 210517_0202_Stats
      Temperatures ok - 210517_0202_Temp
      Bias and leakage ok - 210517_0200_Bias 
02:05 System wide checks:
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x155d : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      8      3      3      3      5      2      2      2      3      7   : 39460
aida02 :     22      4      5      0      3      2      2      3      2      3      7   : 39560
aida03 :     21     10      2      1      1      3      2      4      2      3      7   : 40100
aida04 :     18      7      2      0      3      3      3      2      2      3      7   : 39392
aida05 :     16      6      5      1      2      4      3      2      2      3      7   : 39520
aida06 :     18      6      6      2      2      2      2      3      2      3      7   : 39576
aida07 :     12      8      6      1      2      3      1      3      2      3      7   : 39408
aida08 :     13     11      4      0      1      3      3      2      2      3      7   : 39308
aida09 :     10      8      4      2      3      4      2      3      2      3      7   : 39848
aida10 :     16      8      2      1      2      4      2      3      1      4      7   : 40768
aida11 :     26     10      2      2      1      2      2      3      2      3      7   : 39512
aida12 :     16      9      2      1      2      4      2      2      2      3      7   : 39240
aida13 :     15      9      3      2      2      4      2      2      2      3      7   : 39284
aida14 :     25     10      2      3      3      3      2      3      2      3      7   : 39796
aida15 :      4      2      2      1      1      3      2      2      2      3      7   : 38944
aida16 :     16      7      3      3      1      3      3      2      2      3      7   : 39368

04:00   Statistics ok - 210517_0403_Stats
        Temperatures ok - 210517_0403_Temp
        Bias and leakage ok - 210517_0400_Bias 
04:05 System wide checks:
      Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida06 failed
FEE64 module aida10 failed
Calibration test result: Passed 12, Failed 4

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x1560 : 	 14  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      7      5      2      3      5      2      2      2      3      7   : 39492
aida02 :     23      6      7      2      1      3      2      3      2      3      7   : 39676
aida03 :     17      9      3      2      0      3      2      3      2      3      7   : 39548
aida04 :     16      4      1      2      1      3      3      3      2      3      7   : 39792
aida05 :     22      7      3      3      1      4      3      3      2      3      7   : 40032
aida06 :     18      6      7      3      1      3      2      3      2      3      7   : 39688
aida07 :     18      7      4      1      2      4      1      4      2      3      7   : 40032
aida08 :     19      5      4      1      1      3      3      3      2      3      7   : 39828
aida09 :     10      6      7      1      3      4      2      2      2      3      7   : 39336
aida10 :     12     10      2      3      1      4      2      3      1      4      7   : 40768
aida11 :     10      9      4      3      2      1      2      3      2      3      7   : 39440
aida12 :      8      7      6      1      2      4      2      3      2      3      7   : 39768
aida13 :     11      2      6      1      2      3      2      3      2      3      7   : 39612
aida14 :     27      7      4      2      3      3      2      3      2      3      7   : 39780
aida15 :      4      2      3      1      1      3      2      2      2      3      7   : 38960
aida16 :     18      8      4      2      1      4      2      2      2      3      7   : 39240


06:27 aida07 has dropped out and lost connection with the merger. Powercycling AIDA
06:48 Restarted system but the noise was extremely high. Equivalent to yesterday 15:00. Restarting again.
07:01 it is working fine.
07:40 Screen shots after the restart
 	Statistics ok - 210517_0742_Stats
        Temperatures ok - 210517_0743_Temp
        Bias and leakage ok - 210517_0740_Bias 

System wide checks

	
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0x61da : 	 0x61dd : 	 3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 16, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20      6      3      1      1      3      1      3      3      3      7   : 40336
aida02 :     11      6      0      1      1      5      2      4      2      3      7   : 40252
aida03 :      0      5      2      3      3      3      3      4      2      3      7   : 40424
aida04 :      2      4      2      1      2      4      3      3      2      3      7   : 39912
aida05 :     11      7      5      1      2      2      4      2      3      3      7   : 40532
aida06 :     20      6      0      2      3      2      1      3      3      3      7   : 40320
aida07 :     15      5      1      2      2      3      2      2      3      3      7   : 40116
aida08 :      2      3      4      1      3      3      2      2      3      3      7   : 40128
aida09 :     18      5      1      3      2      4      2      2      3      3      7   : 40288
aida10 :     11      6      3      4      2      3      3      2      2      4      7   : 41484
aida11 :      4      4      5      2      2      2      3      3      2      3      7   : 39744
aida12 :     18      7      2      3      2      2      4      3      2      3      7   : 40064
aida13 :      2      2      1      3      0      3      2      4      2      3      7   : 39944
aida14 :     16      4      5      3      3      3      3      2      2      4      7   : 41552
aida15 :      9      3      1      1      2      3      2      3      3      3      7   : 40556
aida16 :     16      5      1      2      1      2      3      4      2      3      7   : 40184

Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200

*****	08.30	*****

Checked R12_34 through analyser, dead time seems okay:

*** Timestamp elapsed time:      232.154 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.021                0.000
  1                2.781                0.000
  2                0.849                0.000
  3                5.686                0.000
  4                0.000               25.816
  5                0.017                0.000
  6                0.177                0.000
  7                1.349                0.000
  8                1.791                0.000
  9                0.000               91.660
 10                6.194                0.000
 11                0.320                0.000
 12                1.425                0.000
 13                0.000               72.999
 14                0.052                0.000
 15                1.729                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s

*****	09.00	*****

Checked R12_48 through analyser, dead time seems okay:

*** Timestamp elapsed time:      170.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.196                0.000
  1                6.294                0.000
  2                0.992                0.000
  3                8.944                0.000
  4                0.024                0.000
  5                0.001                0.000
  6                0.605                0.000
  7                2.096                0.000
  8                1.654                0.000
  9                0.000               31.395
 10                6.175                0.000
 11                2.524                0.000
 12                2.076                0.000
 13                0.000               10.337
 14                0.103                0.000
 15                2.203                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0902_Stats.png)

*****	09.30	*****

AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20

Ran analyser for R13_16

 *** Timestamp elapsed time:      168.059 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.047                0.000
  1                2.282                0.000
  2                0.080                0.000
  3                0.806                0.000
  4                0.000                0.000
  5                0.260                0.000
  6                0.030                0.000
  7                1.471                0.000
  8               67.467                0.000
  9               80.552                0.000
 10                0.057                0.000
 11                0.000               11.671
 12                1.191                0.000
 13                0.080                0.000
 14                0.091                0.000
 15                0.202                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision

 *** Timestamp elapsed time:      149.082 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                3.580                0.000
  2                0.000                0.000
  3                2.800                0.000
  4                0.000                0.000
  5                0.514                0.000
  6                0.140                0.000
  7                1.372                0.000
  8               55.544                0.000
  9               66.588                0.000
 10                1.065                0.000
 11                0.000               10.010
 12                0.681                0.000
 13                0.044                0.000
 14                0.070                0.000
 15                0.400                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)

Merger rate ~5M items/s
Tape server ~17 MB/s

Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision

*** Timestamp elapsed time:      122.147 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.027                0.000
  1               19.945                0.000
  2                0.601                0.000
  3               21.555                0.000
  4                0.042                0.000
  5                3.226                0.000
  6                1.211                0.000
  7                9.674                0.000
  8               52.817                0.000
  9               54.972                0.000
 10                1.848                0.000
 11                0.008                0.000
 12                2.034                0.000
 13                0.278                0.000
 14                0.018                0.000
 15                0.634                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

high deatime now also seen in aida02 and aida04

Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check

 *** Timestamp elapsed time:      205.006 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.176
  1                0.438                0.000
  2                0.199                0.000
  3                2.927                0.000
  4                0.000              140.347
  5                1.282                0.000
  6                0.150                0.000
  7                2.097                0.000
  8                1.673                0.000
  9                0.000               87.749
 10                1.150                0.000
 11                0.489                0.000
 12                0.374                0.000
 13                0.048               20.995
 14                0.251                0.000
 15                2.441                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000


12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs

Date:17/05/2021
Elog for shift: 08.00 to 12.00
Author: Lewis Sexton
Screenshots saved to: /tmp/Elog_210517_0800_1200

*****	08.30	*****

Checked R12_34 through analyser, dead time seems okay:

*** Timestamp elapsed time:      232.154 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.021                0.000
  1                2.781                0.000
  2                0.849                0.000
  3                5.686                0.000
  4                0.000               25.816
  5                0.017                0.000
  6                0.177                0.000
  7                1.349                0.000
  8                1.791                0.000
  9                0.000               91.660
 10                6.194                0.000
 11                0.320                0.000
 12                1.425                0.000
 13                0.000               72.999
 14                0.052                0.000
 15                1.729                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0833_Stats.png)
Merger rate ~3M items/s
Tape server ~12 MB/s

*****	09.00	*****

Checked R12_48 through analyser, dead time seems okay:

*** Timestamp elapsed time:      170.459 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.196                0.000
  1                6.294                0.000
  2                0.992                0.000
  3                8.944                0.000
  4                0.024                0.000
  5                0.001                0.000
  6                0.605                0.000
  7                2.096                0.000
  8                1.654                0.000
  9                0.000               31.395
 10                6.175                0.000
 11                2.524                0.000
 12                2.076                0.000
 13                0.000               10.337
 14                0.103                0.000
 15                2.203                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

Stats checked seem okay (0902_Stats.png)

*****	09.30	*****

AIDA crash aida05 displaying 0 in stats
Power cycle performed with help of TD
Writing to R13 now
MBS relay restared
AIDA back up at 10.20

Ran analyser for R13_16

 *** Timestamp elapsed time:      168.059 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.047                0.000
  1                2.282                0.000
  2                0.080                0.000
  3                0.806                0.000
  4                0.000                0.000
  5                0.260                0.000
  6                0.030                0.000
  7                1.471                0.000
  8               67.467                0.000
  9               80.552                0.000
 10                0.057                0.000
 11                0.000               11.671
 12                1.191                0.000
 13                0.080                0.000
 14                0.091                0.000
 15                0.202                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

High deadtime in aida09 and aida10, changed slow comparator threshold to 0xe (previously 0xc)
Ran analyser for R13_21 for comparision

 *** Timestamp elapsed time:      149.082 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                3.580                0.000
  2                0.000                0.000
  3                2.800                0.000
  4                0.000                0.000
  5                0.514                0.000
  6                0.140                0.000
  7                1.372                0.000
  8               55.544                0.000
  9               66.588                0.000
 10                1.065                0.000
 11                0.000               10.010
 12                0.681                0.000
 13                0.044                0.000
 14                0.070                0.000
 15                0.400                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE temp slightly high in aida04, 0.44 above limit of 65 (FEETemps_1042.png)

Merger rate ~5M items/s
Tape server ~17 MB/s

Raised slow comparator in aida09 and aida10 to 0xf, analyse R13_26 for comparision

*** Timestamp elapsed time:      122.147 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.027                0.000
  1               19.945                0.000
  2                0.601                0.000
  3               21.555                0.000
  4                0.042                0.000
  5                3.226                0.000
  6                1.211                0.000
  7                9.674                0.000
  8               52.817                0.000
  9               54.972                0.000
 10                1.848                0.000
 11                0.008                0.000
 12                2.034                0.000
 13                0.278                0.000
 14                0.018                0.000
 15                0.634                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

high deatime now also seen in aida02 and aida04

Several powercycles performed as high rates after restart, analysis of R14_13 showed low deadtime for all FEEs
Back running as of 10.45
Analyser of R14_15 to double check

 *** Timestamp elapsed time:      205.006 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000               40.176
  1                0.438                0.000
  2                0.199                0.000
  3                2.927                0.000
  4                0.000              140.347
  5                1.282                0.000
  6                0.150                0.000
  7                2.097                0.000
  8                1.673                0.000
  9                0.000               87.749
 10                1.150                0.000
 11                0.489                0.000
 12                0.374                0.000
 13                0.048               20.995
 14                0.251                0.000
 15                2.441                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000


12:04 During access Sultan added an additional ground cable and also noted the bias cable going into FEE12 was loose. He pushed it in solidly and the rates appear to have dropped across the FEEs


12:52 Check FPGA Timestamp errors gives following error message:

Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.
can't read "TS_Base_Collected": no such variable while executing "if { $TS_Base_Collected == 0 } { collect_TS_Base }" (procedure "do_TS_errors" line 8) invoked from within "do_TS_errors " ("TSERRORS" arm line 1) invoked from within "switch -glob $w { RESET {set started 0} CLEAR {EmptyLog} ELOG {PrintLog} ..." (procedure "do_click" line 23) invoked from within "do_click" invoked from within "if {$started != 0} { variable JS "" variable LogFlag if {$LogFlag == 1} {InsertLog "Last Updated: [clock format [clock seconds] -form..." (file "Check.tcl" line 8) invoked from within "source Check.tcl" (in namespace eval "::Check" script line 6) invoked from within "namespace eval Check { global env source [file join $env(MIDASBASE) TclHttpd tcl Common common.tcl] source [file join $env(MIDASB..." invoked from within "subst { [Doc_Dynamic] <! [global Httpd; upvar #0 Httpd[set Httpd(currentSocket)] data; set ClientIPAddress $data(ipaddr); set MyInfo $data(self)] >..." ("uplevel" body line 1) invoked from within "uplevel #0 [list subst $script]" (procedure "SubstFile" line 12) invoked from within "SubstFile /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml {}" ("uplevel" body line 1) invoked from within "uplevel 1 [list SubstFile $path $interp]" (procedure "Subst_File" line 7) invoked from within "Subst_File $template $interp" invoked from within "TemplateInstantiate $sock $path {} $suffix {} $Template(templateInterp)" (procedure "Doc_application/x-tcl-template" line 9) invoked from within "$cmd $path $suffix $sock" (procedure "Doc_Handle" line 20) invoked from within "Doc_Handle $prefix $path $suffix $sock" (procedure "DocDomain" line 44) invoked from within "DocDomain / /MIDAS/TclHttpd/Html sock8 AIDA/Check/Check.tml" ("eval" body line 1) invoked from within "eval $Url(command,$prefix) [list $sock $suffix]"
Tcl Call Trace
6: DocSubstSystemFile sock8 error 500 can't\ read\ &quot\;TS_Base_Collected&quot\;:\ no\ such\ variable\n\ \ \ \ while\ executing\n&quot\;if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}&quot\;\n\ \ \ \ (procedure\ &quot\;do_TS_errors&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;do_TS_errors\ &quot\;\n\ \ \ \ (&quot\;TSERRORS&quot\;\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...&quot\;\n\ \ \ \ (procedure\ &quot\;do_click&quot\;\ line\ 23)\n\ \ \ \ invoked\ from\ within\n&quot\;do_click&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ &quot\;&quot\;\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ &quot\;Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...&quot\;\n\ \ \ \ (file\ &quot\;Check.tcl&quot\;\ line\ 8)\n\ \ \ \ invoked\ from\ within\n&quot\;source\ Check.tcl&quot\;\n\ \ \ \ (in\ namespace\ eval\ &quot\;::Check&quot\;\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n&quot\;namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;subst\ \{\n\n[Doc_Dynamic\]\n\n\n&lt\;!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n&gt\;...&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ #0\ [list\ subst\ \$script\]&quot\;\n\ \ \ \ (procedure\ &quot\;SubstFile&quot\;\ line\ 12)\n\ \ \ \ invoked\ from\ within\n&quot\;SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}&quot\;\n\ \ \ \ (&quot\;uplevel&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]&quot\;\n\ \ \ \ (procedure\ &quot\;Subst_File&quot\;\ line\ 7)\n\ \ \ \ invoked\ from\ within\n&quot\;Subst_File\ \$template\ \$interp&quot\;\n\ \ \ \ invoked\ from\ within\n&quot\;TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_application/x-tcl-template&quot\;\ line\ 9)\n\ \ \ \ invoked\ from\ within\n&quot\;\$cmd\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;Doc_Handle&quot\;\ line\ 20)\n\ \ \ \ invoked\ from\ within\n&quot\;Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock&quot\;\n\ \ \ \ (procedure\ &quot\;DocDomain&quot\;\ line\ 44)\n\ \ \ \ invoked\ from\ within\n&quot\;DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml&quot\;\n\ \ \ \ (&quot\;eval&quot\;\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n&quot\;eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]&quot\;
5: Doc_Error sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\"
4: Url_Unwind sock8 can't\ read\ \"TS_Base_Collected\":\ no\ such\ variable\n\ \ \ \ while\ executing\n\"if\ \{\ \$TS_Base_Collected\ ==\ 0\ \}\ \{\ collect_TS_Base\ \}\"\n\ \ \ \ (procedure\ \"do_TS_errors\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"do_TS_errors\ \"\n\ \ \ \ (\"TSERRORS\"\ arm\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"switch\ -glob\ \$w\ \{\n\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ RESET\ \ \ \ \{set\ started\ 0\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ CLEAR\ \ \ \ \{EmptyLog\}\n\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ ELOG\ \ \ \ \ \{PrintLog\}\n\ \ \ \ \ \ \ \ \ ...\"\n\ \ \ \ (procedure\ \"do_click\"\ line\ 23)\n\ \ \ \ invoked\ from\ within\n\"do_click\"\n\ \ \ \ invoked\ from\ within\n\"if\ \{\$started\ !=\ 0\}\ \{\n\n\ \ \ \ \ variable\ JS\ \"\"\n\ \ \ \ \ variable\ LogFlag\n\n\ \ \ \ \ if\ \{\$LogFlag\ ==\ 1\}\ \{InsertLog\ \"Last\ Updated:\ [clock\ format\ [clock\ seconds\]\ -form...\"\n\ \ \ \ (file\ \"Check.tcl\"\ line\ 8)\n\ \ \ \ invoked\ from\ within\n\"source\ Check.tcl\"\n\ \ \ \ (in\ namespace\ eval\ \"::Check\"\ script\ line\ 6)\n\ \ \ \ invoked\ from\ within\n\"namespace\ eval\ Check\ \{\n\ \ \ \ \ \ \ global\ env\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASBASE)\ TclHttpd\ tcl\ Common\ common.tcl\]\n\ \ \ \ \ \ \ source\ [file\ join\ \$env(MIDASB...\"\n\ \ \ \ invoked\ from\ within\n\"subst\ \{\n\n[Doc_Dynamic\]\n\n\n<!\n[global\ Httpd\;\ upvar\ #0\ Httpd[set\ Httpd(currentSocket)\]\ data\;\ set\ ClientIPAddress\ \$data(ipaddr)\;\ set\ MyInfo\ \$data(self)\]\n>...\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ #0\ [list\ subst\ \$script\]\"\n\ \ \ \ (procedure\ \"SubstFile\"\ line\ 12)\n\ \ \ \ invoked\ from\ within\n\"SubstFile\ /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119/TclHttpd/Html/AIDA/Check/Check.tml\ \{\}\"\n\ \ \ \ (\"uplevel\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"uplevel\ 1\ [list\ SubstFile\ \$path\ \$interp\]\"\n\ \ \ \ (procedure\ \"Subst_File\"\ line\ 7)\n\ \ \ \ invoked\ from\ within\n\"Subst_File\ \$template\ \$interp\"\n\ \ \ \ invoked\ from\ within\n\"TemplateInstantiate\ \$sock\ \$path\ \{\}\ \$suffix\ \{\}\ \$Template(templateInterp)\"\n\ \ \ \ (procedure\ \"Doc_application/x-tcl-template\"\ line\ 9)\n\ \ \ \ invoked\ from\ within\n\"\$cmd\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"Doc_Handle\"\ line\ 20)\n\ \ \ \ invoked\ from\ within\n\"Doc_Handle\ \$prefix\ \$path\ \$suffix\ \$sock\"\n\ \ \ \ (procedure\ \"DocDomain\"\ line\ 44)\n\ \ \ \ invoked\ from\ within\n\"DocDomain\ /\ /MIDAS/TclHttpd/Html\ sock8\ AIDA/Check/Check.tml\"\n\ \ \ \ (\"eval\"\ body\ line\ 1)\n\ \ \ \ invoked\ from\ within\n\"eval\ \$Url(command,\$prefix)\ [list\ \$sock\ \$suffix\]\" NONE
3: Url_DeferredDispatch / AIDA/Check/Check.tml sock8 buffer {}
2: HttpdReadPost sock8 buffer 16384 {Url_DeferredDispatch / AIDA/Check/Check.tml}
1: HttpdReadPostGlobal sock8 Httpds

Dead time check:

[npg@aidas-gsi S496]$ ~/analyser/analyser R14_24
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252995858 (  451517.9 Hz)
        Other data format:    8924143 (   15926.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         85 (       0.2 Hz)
                               RESUME:         85 (       0.2 Hz)
                              SYNC100:      34087 (      60.8 Hz)
                              WR48-63:      34087 (      60.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    8855799 (   15804.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      560.323 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000              349.960
  1                0.002                0.000
  2                0.000                5.974
  3                1.105                0.000
  4                0.000              447.454
  5                0.000                2.809
  6                0.000               16.727
  7                0.137                0.000
  8                0.090                0.000
  9                0.000              429.897
 10                0.461                0.000
 11                0.018                0.000
 12                0.000                8.985
 13                0.000              466.311
 14                0.000               15.095
 15                0.346                0.000

13:24 Beam is off at the moment...

13:43 Changing spill structure from 1.5s/3.5s to 1s. 

13:50

[npg@aidas-gsi S496]$ ~/analyser/analyser R14_29
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  251763810 (  930342.1 Hz)
        Other data format:   10156190 (   37530.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        205 (       0.8 Hz)
                               RESUME:        205 (       0.8 Hz)
                              SYNC100:      33009 (     122.0 Hz)
                              WR48-63:      33009 (     122.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:   10089762 (   37284.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     527077 (    1947.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      270.614 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.034                6.803
  1                8.461                0.000
  2                0.569                0.000
  3               11.664                0.000
  4                0.024               26.416
  5                0.854                0.000
  6                0.069                0.000
  7                0.741                0.000
  8                1.336                0.000
  9                0.000               75.926
 10                0.353                0.000
 11                0.105                0.000
 12                0.070                0.000
 13                0.000               93.585
 14                0.017                0.000
 15                0.121                0.000

15:06 Spill structure back to 1.5 on 2s off.
Attachment 1: 210517_0200_Bias.png
210517_0200_Bias.png
Attachment 2: 210517_0202_stats.png
210517_0202_stats.png
Attachment 3: 210517_0202_temp.png
210517_0202_temp.png
Attachment 4: 210517_0400_Bias.png
210517_0400_Bias.png
Attachment 5: 210517_0403_stats.png
210517_0403_stats.png
Attachment 6: 210517_0403_temp.png
210517_0403_temp.png
Attachment 7: 210517_0740_Bias.png
210517_0740_Bias.png
Attachment 8: 210517_0742_Stats.png
210517_0742_Stats.png
Attachment 9: 210517_0743_Temp.png
210517_0743_Temp.png
Attachment 10: 0833_Stats.png
0833_Stats.png
Attachment 11: 0902_Stats.png
0902_Stats.png
Attachment 12: 1241_Stats.png
1241_Stats.png
Attachment 13: 1347_Stats.png
1347_Stats.png
Attachment 14: FEETemps_1042.png
FEETemps_1042.png
Attachment 15: FEETemps_1246.png
FEETemps_1246.png
Attachment 16: FEETemps_1350.png
FEETemps_1350.png
Attachment 17: leakage_currents_1302.png
leakage_currents_1302.png
  329   Wed May 19 09:09:27 2021 MS, JS, OHMay 16 08:00-24:00
16th May 08:00 - 12:00 shift
Author: MS


08:00


	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



	
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7163 : 	 23  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      7      2      3      2      3      2      4      2      3      7   : 40228
aida02 :      3      8      3      2      2      2      2      4      2      3      7   : 39996
aida03 :     23      9      6      1      0      3      2      4      2      3      7   : 40100
aida04 :     34     27     17      7      2      4      3      3      2      2      7   : 38608
aida05 :     19      9      5      2      2      2      3      3      2      3      7   : 39844
aida06 :      5      5      2      1      0      4      2      4      2      3      7   : 40060
aida07 :     28      2     10      3      2      4      1      4      2      3      7   : 40192
aida08 :     19      5      5      1      2      5      1      3      2      3      7   : 39652
aida09 :     20      8      3      3      1      3      2      3      2      3      7   : 39648
aida10 :     27     10      0      2      2      2      2      3      1      4      7   : 40572
aida11 :      3      3      2      1      2      4      2      3      2      3      7   : 39652
aida12 :     16      7     11      2      3      4      2      2      2      3      7   : 39464
aida13 :     14     10      4      3      1      5      2      2      2      3      7   : 39400
aida14 :     21      9     10      1      1      2      2      3      1      4      7   : 40604
aida15 :     19      8      2      3      0      4      3      2      2      3      7   : 39436
aida16 :     24      5      8      3      3      4      2      2      2      3      7   : 39464







 *** Timestamp elapsed time:      225.065 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.038                0.000
  1                9.479                0.000
  2                0.195                0.000
  3                5.921                0.000
  4                0.000               11.742
  5                0.036                0.000
  6                0.013                0.000
  7                0.498                0.000
  8                0.436                0.000
  9                0.000              107.300
 10                2.787                0.000
 11                0.905                0.000
 12                0.831                0.000
 13                0.000               55.939
 14                0.080                0.000
 15                0.267                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000






10:00


	
FEE64 module aida06 global clocks failed, 6
 FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 14, Failed 2

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



	
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7166 : 	 26  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19      7      6      2      2      3      2      3      2      3      7   : 39716
aida02 :      8      4      3      2      2      3      2      3      2      3      7   : 39600
aida03 :     23     11      5      1      0      4      2      3      1      4      7   : 40740
aida04 :     42     25     16      7      2      4      4      3      2      2      7   : 38864
aida05 :     24      5      5      1      2      2      3      3      1      4      7   : 40824
aida06 :     11      4      4      1      1      4      2      4      2      3      7   : 40172
aida07 :     21      8      5      1      2      3      2      4      2      3      7   : 40196
aida08 :     15      5      6      1      1      4      2      4      2      3      7   : 40228
aida09 :     15     10      4      1      2      3      2      3      2      3      7   : 39660
aida10 :     23      8      2      2      2      2      2      3      1      4      7   : 40572
aida11 :      6      4      4      2      2      4      2      3      2      3      7   : 39736
aida12 :     18      8      8      1      4      4      3      2      2      3      7   : 39720
aida13 :     23      6      2      3      2      5      2      2      2      3      7   : 39436
aida14 :     29      3     11      1      1      2      2      3      1      4      7   : 40604
aida15 :     29      7      2      2      0      4      3      3      2      3      7   : 39948
aida16 :      2      3      6      2      3      4      2      2      2      3      7   : 39296






*** Timestamp elapsed time:      225.065 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.038                0.000
  1                9.479                0.000
  2                0.195                0.000
  3                5.921                0.000
  4                0.000               11.742
  5                0.036                0.000
  6                0.013                0.000
  7                0.498                0.000
  8                0.436                0.000
  9                0.000              107.300
 10                2.787                0.000
 11                0.905                0.000
 12                0.831                0.000
 13                0.000               55.939
 14                0.080                0.000
 15                0.267                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

12:00-16:00
16th May 12:00 - 16:00 shift
Author: JS

11:57 Taking over from Magda. Running full checks.
usbec ok. Max ~1700 Hz 1MHz on DSSD1, DSSD ~ 75%
Current ok  06.410 uA      006.835 uA
Stats good 1Statistics aidas-gsi(6).png
Temps ok 1Temperature and status scan aidas-gsi(6).png
Analysis ok R7_385. Dead time FEE1 a little hight 6%

PAUSE: 166  RESUME: 166

 *** Timestamp elapsed time:      196.305 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.044                0.000
  1               12.405                0.000
  2                0.807                0.000
  3                7.260                0.000
  4                0.014                0.000
  5                0.458                0.000
  6                0.027                0.000
  7                0.497                0.000
  8                0.723                0.000
  9                0.000               88.857
 10                6.189                0.000
 11                1.443                0.000
 12                0.474                0.000
 13                0.000               35.565
 14                0.000                0.000
 15                0.147                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 15, Failed 1

FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x716e : 	 34  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0xa : 	 10  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20      7      4      2      2      3      2      3      1      4      7   : 40712
aida02 :     25     10      7      1      1      4      1      3      2      3      7   : 39556
aida03 :     22     10      5      1      0      4      2      4      2      3      7   : 40216
aida04 :     40     24     17      7      2      5      3      3      2      2      7   : 38736
aida05 :      4      8      3      0      1      3      3      3      1      4      7   : 40768
aida06 :     21      6      6      2      2      3      2      4      2      3      7   : 40228
aida07 :     25      4      6      3      2      3      2      4      2      3      7   : 40260
aida08 :     19      9      4      1      1      4      2      4      2      3      7   : 40244
aida09 :     16      9      4      2      2      3      2      3      2      3      7   : 39688
aida10 :     19     10      5      1      2      2      2      4      1      4      7   : 41100
aida11 :     21     10      2      1      2      3      3      3      2      3      7   : 39908
aida12 :     25      7      7      2      2      3      2      3      2      3      7   : 39756
aida13 :     13      7      3      4      2      5      2      3      2      3      7   : 39964
aida14 :     21      7      9      2      1      2      2      4      1      4      7   : 41116
aida15 :     23      6      2      3      0      4      3      3      2      3      7   : 39948
aida16 :      9     11     10      2      3      4      2      2      2      3      7   : 39452

Tom says Aida09 clock fail is ok as its status bit is "6".
The large white difference for FEE5 is known and has been determined to be ok, a post run investigation will be undertaken.

12:35 -
usbec ok.
Current ok 
Stats good 
Temps ok 
Analysis ok R7_395. Dead time FEE1 still high 6.6%

13:33 -
usbec ok. Max implants ~ 1.8kHz
Current ok    006.850 uA      007.250 uA 
Stats - Aida11 runing low < 5k was ~20k overnight
Temps ok 
Analysis R7_415. Dead time FEE1 & FEE10 high 10%

14:00
usbec ok - ucesb1.png
Current ok - bias1.png
Stast -  Aida11 low - 

14:20 aida fee rebooted itself. A powercycle was performed. Upon reboot we are seeing extremely large amounts of noise in the FEEs. Looking at the waveforms we have very large 100kHz pick up in the FEEs. This has resulted in 50% deadtime in many FEEs including the p+n.

15:28 Because of extremely high rates across all FEEs have decided to do a powercycle. Before restarting the FEEs will give them a couple of minutes to cool.

18:00 Since the start of the shift we have been trying to recover the system froma large increase in noise following the crash at ~14:00.
      During this time NH has entered the area and inspected the system and also grounded the AIDA snout. This provided us with some improvement on the noise.
      The rates are still slightly above where we were before the crash but now appear stable. To counteract the dead time in the n+n strips we have raised the threshold to 0x64 for ASIC4 in all FEEs.
      We are now running with around 10% deadtime on FEE4 and less elsewhere for n+n. For p+n most have zero dead time apart from FEE11 which is still noisy.
      During the time we were trying to recover the system screenshots were taken of the waveforms. He it could be seen that the 100kHz noise was very apparent. Particularly in the n+n strips.

18:08 System wide checks all ok - bar some ADC but waveforms disabled
      Statistics ok - 210516_1809_Stats
      Temperatures ok - 210516_1809_Temp
      Bias and leakage ok - 210516_1810_Bias

18:37 Performed an ASIC check and now the rates have dropped in all n+n strips. Currently very small amounts of dead time
18:40 Realised this was because it raised the threshold of all strips to 0x64 on the n+n side.

19:25 Removed S452 from 1e2.... drive. Before removing checked with Nic backed up to Lustre. Also verified four ourselves.
      Now have around 4.2TB left which will provide around 80 hours of writing

20:16 We noticed iptraf was using around 30% CPU usage. We investigated whether it had any effect on the dead time but from what we have seen it has not.

20:57 System wide checks. Clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x1556 : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Statistics ok - 210516_2056_Stats
      Temp ok -  210516_2058_Temp
      Bias and leakage current ok -  210516_2058_Bias

23:16 System wide checks:
      Clock still ok 
	
		 Base 		Current 	Difference
aida05 fault 	 0x1552 : 	 0x155a : 	 8  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

       Statistics - 210516_2315_Stats
       Temperature - 210516_2316_Temp
       Bias and leakage current ok - 210516_231
Attachment 1: Screenshot_2021-05-16_Statistics_aidas-gsi(4).png
Screenshot_2021-05-16_Statistics_aidas-gsi(4).png
Attachment 2: Screenshot_2021-05-16_Statistics_aidas-gsi(5).png
Screenshot_2021-05-16_Statistics_aidas-gsi(5).png
Attachment 3: Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 4: Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(5).png
Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(5).png
Attachment 5: 1Statistics_aidas-gsi(6).png
1Statistics_aidas-gsi(6).png
Attachment 6: 1Temperature_and_status_scan_aidas-gsi(6).png
1Temperature_and_status_scan_aidas-gsi(6).png
Attachment 7: Bias1.png
Bias1.png
Attachment 8: 210516_0857_Temp.png
210516_0857_Temp.png
Attachment 9: 210516_1554_Layout8.png
210516_1554_Layout8.png
Attachment 10: 210516_1555_Layout7.png
210516_1555_Layout7.png
Attachment 11: 210516_1600_Layout1.png
210516_1600_Layout1.png
Attachment 12: 210516_1601_Stats.png
210516_1601_Stats.png
Attachment 13: 210516_1605_1606.png
210516_1605_1606.png
Attachment 14: 210516_1808_Stats.png
210516_1808_Stats.png
Attachment 15: 210516_1809_Temp.png
210516_1809_Temp.png
Attachment 16: 210516_1810_Bias.png
210516_1810_Bias.png
Attachment 17: 210516_2056_Stats.png
210516_2056_Stats.png
Attachment 18: 210516_2058_Bias.png
210516_2058_Bias.png
Attachment 19: 210516_2313_Stats.png
210516_2313_Stats.png
Attachment 20: 210516_2315_Temp.png
210516_2315_Temp.png
Attachment 21: 210516_2316_Bias.png
210516_2316_Bias.png
  328   Wed May 19 09:06:58 2021 CAMay 16th 00:00 - 08:00
16th May 2021 - 00:00 - 08:00 Shift
Author: CA

00:15 System wide checks;

FEE64 module aida06 global clocks failed, 6
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 14, Failed 2

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x7154 : 	 8  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     19     10      3      1      2      3      2      3      2      3      7   : 39660
aida02 :     29      5      3      1      1      3      2      3      2      3      7   : 39596
aida03 :     15      7      1      1      2      3      2      3      2      3      7   : 39588
aida04 :     44     26     19      5      2      4      3      3      2      2      7   : 38608
aida05 :     20     10      3      0      1      4      2      2      3      3      7   : 40208
aida06 :     27      6      2      1      2      4      2      4      2      3      7   : 40284
aida07 :     25      5      3      1      2      3      2      3      2      3      7   : 39644
aida08 :     23      8      2      3      1      4      2      3      2      3      7   : 39772
aida09 :     23     10      3      2      3      4      1      3      2      3      7   : 39644
aida10 :     18     10      3      2      3      2      2      2      2      4      7   : 41160
aida11 :     16      7      3      2      1      2      2      3      2      3      7   : 39464
aida12 :     21      4      7      1      3      3      3      3      2      3      7   : 40004
aida13 :     25      6      3      2      1      3      1      4      2      3      7   : 39876
aida14 :     20      8     10      1      1      2      2      2      2      4      7   : 41104
aida15 :     26      6      3      3      0      4      3      2      2      3      7   : 39464
aida16 :      9      8      2      4      1      3      1      4      2      3      7   : 39876

00:30 Stats ok - 210516_0030_stats.png
      Temps ok - 210516_0030_temps.png
      Bias ok - 210516_0030_bias.png

00:38 analyzer output R7_178 - Deadtime at 7.3% - 210516_R7_178_analysis.txt

02:21 system wide checks as above, except;

      		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x7159 : 	 13  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x7 : 	 7  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:28 Stats ok - 210516_0230_stats.png
      Temps ok - 210516_0230_temps.png
      Bias ok - 210516_0230_bias.png

02:33 analyzer output R7_216 - Deadtime at 6.5% - 210516_R7_216_analysis.txt

04:36 system wide checks same as above, except;

		 Base 		Current 	Difference
aida05 fault 	 0x714c : 	 0x715c : 	 16  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x9 : 	 9  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
aida13 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

04:37 Stats ok - 210516_0430_stats.png
      Temps ok - 210516_0430_temps.png
      Bias ok - 210516_0430_bias.png

04:44 analyzer output R7_260 - Deadtime at 6.9% - 210516_R7_260_analysis.txt

06:38 system wide checks same as before, except;

		 Base 		Current 	Difference
aida01 fault 	 0xf294 : 	 0xf296 : 	 2  
aida02 fault 	 0xd8ec : 	 0xd8ee : 	 2  
aida03 fault 	 0xf001 : 	 0xf003 : 	 2  
aida04 fault 	 0xd992 : 	 0xd994 : 	 2  
aida05 fault 	 0x714c : 	 0x7161 : 	 21  
aida06 fault 	 0x5a49 : 	 0x5a4a : 	 1  
aida07 fault 	 0x5aca : 	 0x5acb : 	 1  
aida08 fault 	 0xb92e : 	 0xb92f : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

06:40 Stats ok - 210516_0640_stats.png
      Temps ok - 210516_0640_temps.png
      Bias ok - 210516_0640_bias.png

06:45 analyzer output R7_298 - Deadtime at 5.8% - 210516_R7_298_analysis.txt
Attachment 1: 210516_0030_bias.png
210516_0030_bias.png
Attachment 2: 210516_0030_stats.png
210516_0030_stats.png
Attachment 3: 210516_0030_temps.png
210516_0030_temps.png
Attachment 4: 210516_0230_bias.png
210516_0230_bias.png
Attachment 5: 210516_0230_stats.png
210516_0230_stats.png
Attachment 6: 210516_0230_temps.png
210516_0230_temps.png
Attachment 7: 210516_0430_bias.png
210516_0430_bias.png
Attachment 8: 210516_0430_stats.png
210516_0430_stats.png
Attachment 9: 210516_0430_temps.png
210516_0430_temps.png
Attachment 10: 210516_0640_bias.png
210516_0640_bias.png
Attachment 11: 210516_0640_stats.png
210516_0640_stats.png
Attachment 12: 210516_0640_temps.png.png
210516_0640_temps.png.png
Attachment 13: analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255660963 ( 1413413.6 Hz)
        Other data format:    6259037 (   34602.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        192 (       1.1 Hz)
                               RESUME:        192 (       1.1 Hz)
                              SYNC100:      32674 (     180.6 Hz)
                              WR48-63:      32674 (     180.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    6193305 (   34239.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     287054 (    1587.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      180.882 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.117                0.000
  1                8.836                0.000
  2                1.027                0.000
  3                5.545                0.000
  4                0.000                5.604
  5                0.873                0.000
  6                0.370                0.000
  7                0.861                0.000
  8                0.767                0.000
  9                0.000               63.434
 10               13.201                0.000
 11                3.930                0.000
 12                0.813                0.000
 13                0.000               30.131
 14                0.448                0.000
 15                1.895                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9469697       3614          0          0          3          3       1264       1264          0       1080          0      15256
  1   23268933       6078          0          0         41         41       2998       2998          0          0          0      31257
  2   14135373    1756135          0          0          5          5       1968       1968          0    1752189          0      25654
  3   26646926    1718468          0          0         28         28       3507       3507          0    1711398          0      36818
  4    2828773     749772          0          0          0          0        426        426          0     748920          0       7104
  5   14183721     453202          0          0          5          5       1830       1830          0     449532          0      20813
  6   11921192     886700          0          0          5          5       1583       1583          0     883524          0      13410
  7   19791515     651950          0          0          5          5       2639       2639          0     646662          0      25026
  8   22244739       5464          0          0          4          4       2728       2728          0          0          0      32840
  9    1601175        378          0          0          0          0        189        189          0          0          0       2522
 10   41869813      10332          0          0         63         63       5103       5103          0          0          0      18083
 11   18332381       4602          0          0         16         16       2285       2285          0          0          0       5732
 12   14937821       3786          0          0          7          7       1886       1886          0          0          0      28026
 13    2220427        536          0          0          0          0        268        268          0          0          0       2121
 14   11574320       2908          0          0          3          3       1451       1451          0          0          0      17899
 15   20634157       5112          0          0          7          7       2549       2549          0          0          0       4493
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   47.672s (  671.255 blocks/s,  41.953 Mb/s)
Attachment 14: analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255713262 ( 1398395.0 Hz)
        Other data format:    6206738 (   33942.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        185 (       1.0 Hz)
                               RESUME:        185 (       1.0 Hz)
                              SYNC100:      32681 (     178.7 Hz)
                              WR48-63:      32681 (     178.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    6141006 (   33582.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     279005 (    1525.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      182.862 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.488                0.000
  1                8.465                0.000
  2                0.216                0.000
  3                7.120                0.000
  4                0.011                8.896
  5                0.764                0.000
  6                0.401                0.000
  7                1.395                0.000
  8                0.721                0.000
  9                0.000               86.292
 10               11.881                0.000
 11                1.912                0.000
 12                0.187                0.000
 13                0.000               15.181
 14                0.528                0.000
 15                0.095                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9424415       3624          0          0          2          2       1264       1264          0       1092          0      14964
  1   23109905       5922          0          0         39         39       2922       2922          0          0          0      29876
  2   14322958    1782809          0          0          3          3       2029       2029          0    1778745          0      25108
  3   26234410    1716299          0          0         35         35       3458       3458          0    1709313          0      35100
  4    2817740     734656          0          0          1          1        433        433          0     733788          0       6814
  5   13978921     443934          0          0          4          4       1871       1871          0     440184          0      20242
  6   11924553     862150          0          0          4          4       1621       1621          0     858900          0      13000
  7   19324336     623898          0          0         11         11       2446       2446          0     618984          0      24043
  8   21963572       5366          0          0          5          5       2678       2678          0          0          0      32304
  9    1580877        372          0          0          0          0        186        186          0          0          0       2567
 10   42874488      10618          0          0         59         59       5250       5250          0          0          0      17549
 11   18504188       4566          0          0         14         14       2269       2269          0          0          0       5838
 12   14580871       3684          0          0          2          2       1840       1840          0          0          0      27687
 13    2240991        612          0          0          0          0        306        306          0          0          0       2138
 14   11530867       2858          0          0          4          4       1425       1425          0          0          0      17152
 15   21300170       5370          0          0          2          2       2683       2683          0          0          0       4623
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   45.405s (  704.764 blocks/s,  44.048 Mb/s)
Attachment 15: analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255591110 ( 1400708.0 Hz)
        Other data format:    6328890 (   34684.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        179 (       1.0 Hz)
                               RESUME:        179 (       1.0 Hz)
                              SYNC100:      32680 (     179.1 Hz)
                              WR48-63:      32680 (     179.1 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    6263172 (   34323.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     288458 (    1580.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      182.473 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                7.649                0.000
  2                0.265                0.000
  3                5.929                0.000
  4                0.000               21.293
  5                1.614                0.000
  6                0.390                0.000
  7                0.004                0.000
  8                1.749                0.000
  9                0.000               75.748
 10               12.608                0.000
 11                1.662                0.000
 12                0.334                0.000
 13                0.000               39.700
 14                0.092                0.000
 15                1.009                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9481493       3580          0          0          0          0       1244       1244          0       1092          0      15504
  1   23234003       5924          0          0         42         42       2920       2920          0          0          0      30950
  2   14360490    1787895          0          0          2          2       2026       2026          0    1783839          0      26238
  3   26580599    1734745          0          0         32         32       3537       3537          0    1727607          0      36830
  4    2867864     759325          0          0          0          0        449        449          0     758427          0       7075
  5   13998560     454755          0          0          6          6       1806       1806          0     451131          0      20732
  6   12019427     893028          0          0          3          3       1644       1644          0     889734          0      13550
  7   19745451     656494          0          0          1          1       2575       2575          0     651342          0      25286
  8   21649633       5478          0          0          7          7       2732       2732          0          0          0      32719
  9    1602793        374          0          0          0          0        187        187          0          0          0       2586
 10   42457575      10646          0          0         62         62       5261       5261          0          0          0      18276
 11   18423606       4634          0          0         16         16       2301       2301          0          0          0       6050
 12   14493303       3540          0          0          2          2       1768       1768          0          0          0      27878
 13    2240564        512          0          0          0          0        256        256          0          0          0       2153
 14   11446056       2794          0          0          2          2       1395       1395          0          0          0      17997
 15   20989693       5166          0          0          4          4       2579       2579          0          0          0       4634
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   52.041s (  614.900 blocks/s,  38.431 Mb/s)
Attachment 16: analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  254484788 ( 1267953.1 Hz)
        Other data format:    7435212 (   37045.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        166 (       0.8 Hz)
                               RESUME:        166 (       0.8 Hz)
                              SYNC100:      32747 (     163.2 Hz)
                              WR48-63:      32747 (     163.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    7369386 (   36717.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     359272 (    1790.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      200.705 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.411                0.000
  1               11.545                0.000
  2                1.046                0.000
  3                6.943                0.000
  4                0.008                8.993
  5                0.438                0.000
  6                0.000                0.000
  7                1.702                0.000
  8                0.655                0.000
  9                0.000               75.734
 10                6.237                0.000
 11                1.119                0.000
 12                0.217                0.000
 13                0.000               28.278
 14                0.250                0.000
 15                0.105                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8401665       3348          0          0          2          2       1072       1072          0       1200          0      19564
  1   26165949       6958          0          0         57         57       3422       3422          0          0          0      37826
  2   13624578    2004081          0          0          4          4       1973       1973          0    2000127          0      32349
  3   29939387    1947767          0          0         35         35       4004       4004          0    1939689          0      45332
  4    3232848     946545          0          0          1          1        548        548          0     945447          0       8745
  5   15270389     573894          0          0          6          6       1986       1986          0     569910          0      26107
  6   10372112    1114231          0          0          0          0       1457       1457          0    1111317          0      17063
  7   19130864     806574          0          0          8          8       2431       2431          0     801696          0      31336
  8   24018310       5808          0          0          5          5       2899       2899          0          0          0      41117
  9    1787485        418          0          0          0          0        209        209          0          0          0       3357
 10   42125761      10654          0          0         29         29       5298       5298          0          0          0      22828
 11   16187533       3960          0          0         11         11       1969       1969          0          0          0       7397
 12   14311704       3490          0          0          1          1       1744       1744          0          0          0      35224
 13    2388212        620          0          0          0          0        310        310          0          0          0       2664
 14    9736084       2414          0          0          1          1       1206       1206          0          0          0      22553
 15   17791907       4450          0          0          6          6       2219       2219          0          0          0       5810
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   43.287s (  739.250 blocks/s,  46.203 Mb/s)
  327   Wed May 19 06:59:33 2021 OHWednesday 19th May 08:00 - 16:00
08:00 OH Takes over for CA
      Thresholds 0xc for p+n and 0x19 for n+n ASICs 1-3 and 0x64 for n+nASIC4

 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

       Max average dead time in R14_586 3.2% which is 9.6% on spill
       Stable from yesterday

10:02 System wide checks. All ok expect WR
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x554 : 	 84  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics - Attachment 1
      Temp - attachment 2
      Bias and leakage currents ok - Attachment 3


11:17 A new run was started as the microspill has been changed. I believe to optimise FRS deadtime.
11:45 They are still working on it

12:05 Beam off for a bit while they optimise spill

12:15 System wide checks ok expect WR
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x567 : 	 103  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics - Attachment 4
     Temp - Attachment 5
     Bias - Attachment 6

13:11 Magda has said that the changes to the spill structure only gained them a few percent in dead time
      Further changes could be done, but would take at least two hours

13:19 Dead time following the current spill changes
	R14_648	R14_649	R14_650	R14_651	Sum DT (S)   Avergage DT (%)   On Spill DT (%)
FEE	244.823	252.168	247.155	240.421	984.567		
0	0.063	0.025	0.105	0.096	0.289	     0.029353005	0.088059015
1	3.981	4.907	4.17	2.484	15.542	     1.578561947	4.73568584
2	0.01	0	0.121	0.779	0.91	     0.092426417	0.277279251
3	3.625	2.972	2.766	3.335	12.698	     1.289704002	3.869112006
4	0.088	0.1	0.12	0.136	0.444	     0.045095966	0.135287898
5	9.392	5.798	9.835	8.01	33.035	     3.355282068	10.0658462
6	0.153	0.075	0.01	0.517	0.755	     0.076683456	0.230050367
7	7.4	7.565	7.703	6.415	29.083	     2.953887343	8.86166203
8	0.221	0.606	1.321	0.539	2.687	     0.272911849	0.818735546
9	0	0	0	0	0	     0          	0
10	1.381	0.368	1.196	1.091	4.036	     0.409926394	1.229779182
11	0.066	0.025	0.039	0.017	0.147	     0.014930421	0.044791264
12	0.166	0.041	0.062	0.024	0.293	     0.029759275	0.089277825
13	0	0	0	0	0	     0          	0
14	0.71	0	0.005	0.074	0.789	     0.08013675	        0.240410251
15	0.239	0.17	0.208	0.286	0.903	     0.091715444	0.275146333
16	0	0	0	0	0	     0          	0

14:03 System wide checks ok - WR same difference as before
      Statistics - attachment 7
      Temperatures - attachment 8
      Bias and leakage currents - attachment 9
Attachment 1: 210519_1002_Stats.png
210519_1002_Stats.png
Attachment 2: 210519_1002_Temp.png
210519_1002_Temp.png
Attachment 3: 210519_1003_Bias.png
210519_1003_Bias.png
Attachment 4: 210519_1213_Stats.png
210519_1213_Stats.png
Attachment 5: 210519_1214_Temp.png
210519_1214_Temp.png
Attachment 6: 210519_1215_Bioas.png
210519_1215_Bioas.png
Attachment 7: 210519_1403_Stats.png
210519_1403_Stats.png
Attachment 8: 210519_1404_Temp.png
210519_1404_Temp.png
Attachment 9: 210519_1405_bias.png
210519_1405_bias.png
  326   Tue May 18 23:11:40 2021 CAWednesday May 19th 00:00 - 08:00
00:00 CA takes over
      LS has performed usual checks, see previous Elog entry - all ok

00:30 stats ok
      R14_470 deadtime at 4.4%

02:10 system wide checks ok except:

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x52f : 	 47  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

02:12 FEE64 Temps ok - attachment 1
      Stats ok - attachment 2
      detector bias / leakage currents ok - attachment 3
      dead-times ok - R14_497 max 4% - attachment 4

03:27 DAQ continues ok - writing to file R14_517

04:07 system wide checks ok except:

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x537 : 	 55  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      FEE64 Temps ok - attachment 6
      Stats ok - attachment 5
      detector bias / leakage currents ok - attachment 7
      dead-times ok - R14_526 max 4% - attachment 8

05:27 bad timestamp error in merger terminal - attachment 9

06:05 all system wide checks ok except;

      all ADC fail calibration (expected, no waveforms)

      WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x53d : 	 61  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

06:07 FEE64 Temps ok - attachment 10
      Stats ok - attachment 11
      detector bias / leakage currents ok - attachment 12
      dead-times ok - R14_558 max 3.6% - attachment 13

07:01 another bad timestamp error in merger terminal - attachment 14
Attachment 1: Screenshot_from_2021-05-19_01-13-48.png
Screenshot_from_2021-05-19_01-13-48.png
Attachment 2: Screenshot_from_2021-05-19_01-14-09.png
Screenshot_from_2021-05-19_01-14-09.png
Attachment 3: Screenshot_from_2021-05-19_01-14-24.png
Screenshot_from_2021-05-19_01-14-24.png
Attachment 4: Screenshot_from_2021-05-19_01-19-37.png
Screenshot_from_2021-05-19_01-19-37.png
Attachment 5: Screenshot_from_2021-05-19_03-04-39.png
Screenshot_from_2021-05-19_03-04-39.png
Attachment 6: Screenshot_from_2021-05-19_03-05-33.png
Screenshot_from_2021-05-19_03-05-33.png
Attachment 7: Screenshot_from_2021-05-19_03-05-53.png
Screenshot_from_2021-05-19_03-05-53.png
Attachment 8: Screenshot_from_2021-05-19_03-09-03.png
Screenshot_from_2021-05-19_03-09-03.png
Attachment 9: Screenshot_from_2021-05-19_04-26-09.png
Screenshot_from_2021-05-19_04-26-09.png
Attachment 10: Screenshot_from_2021-05-19_05-08-01.png
Screenshot_from_2021-05-19_05-08-01.png
Attachment 11: Screenshot_from_2021-05-19_05-08-23.png
Screenshot_from_2021-05-19_05-08-23.png
Attachment 12: Screenshot_from_2021-05-19_05-08-46.png
Screenshot_from_2021-05-19_05-08-46.png
Attachment 13: Screenshot_from_2021-05-19_05-11-33.png
Screenshot_from_2021-05-19_05-11-33.png
Attachment 14: Screenshot_from_2021-05-19_06-12-29.png
Screenshot_from_2021-05-19_06-12-29.png
  325   Tue May 18 18:47:25 2021 LSTuesday 18th May 20:00 - 24:00
19.50
Stats okay (Fig 1)
FEE temps okay (Fig 2)
Bias okay (Fig 3)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x520 : 	 32  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~1.8M items/s
Tape server ~7MB/s

Analysed R14_399, deadtimes are okay (Fig 4)

20.30
Stats okay (Fig 5)
Analysed R14_407, deadtimes are okay (Fig 6)

21.00
Stats okay (Fig 7)
Analysed R14_415, deadtimes are okay (Fig 8)

21.30
Stats okay (Fig 9)
Analysed R14_422, deadtimes are okay (Fig 10)

22.00
Stats okay (Fig 11)
FEE temps okay (Fig 12)
Bias okay (Fig 13)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x524 : 	 36  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Merger rate ~2.3M items/s
Tape server ~8MB/s

Analysed R14_430, deadtimes are okay (Fig 14)

22.30
Stats okay (Fig 15)
Analysed R14_438, deadtimes are okay (Fig 16)
FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5%

23.00
Stats okay (Fig 17)
Analysed R14_445, deadtimes are okay (Fig 18)

23.30
Stats okay (Fig 19)
Analysed R14_454, deadtimes are okay (Fig 20)

24.00
Stats okay (Fig 21)
FEE temps okay (Fig 22)
Bias okay (Fig 23)

System wide checks okay except:

		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x529 : 	 41  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

The difference has increased over time (not sure if an issue)

Merger rate ~3M items/s
Tape server ~8MB/s

Analysed R14_462, deadtimes are okay (Fig 24)
Attachment 1: 210518_Stats_1950.png
210518_Stats_1950.png
Attachment 2: 210518_FEEtemps_1950.png
210518_FEEtemps_1950.png
Attachment 3: 210518_Bias_1950.png
210518_Bias_1950.png
Attachment 4: 210518_Deadtimes_R14_399.png
210518_Deadtimes_R14_399.png
Attachment 5: 210518_Stats_2028.png
210518_Stats_2028.png
Attachment 6: 210518_Deadtimes_R14_407.png
210518_Deadtimes_R14_407.png
Attachment 7: 210518_Stats_2059.png
210518_Stats_2059.png
Attachment 8: 210518_Deadtimes_R14_415.png
210518_Deadtimes_R14_415.png
Attachment 9: 210518_Stats_2130.png
210518_Stats_2130.png
Attachment 10: 210518_Deadtimes_R14_422.png
210518_Deadtimes_R14_422.png
Attachment 11: 210518_Stats_2158.png
210518_Stats_2158.png
Attachment 12: 210518_FEEtemps_2158.png
210518_FEEtemps_2158.png
Attachment 13: 210518_bias_2158.png
210518_bias_2158.png
Attachment 14: 210518_Deadtimes_R14_430.png
210518_Deadtimes_R14_430.png
Attachment 15: 210518_Stats_2230.png
210518_Stats_2230.png
Attachment 16: 210518_deadtimes_R14_438.png
210518_deadtimes_R14_438.png
Attachment 17: 210518_Stats_2300.png
210518_Stats_2300.png
Attachment 18: 210518_Deadtimes_R14_445.png
210518_Deadtimes_R14_445.png
Attachment 19: 210518_Stats_2330.png
210518_Stats_2330.png
Attachment 20: 210518_Deadtimes_R14_454.png
210518_Deadtimes_R14_454.png
Attachment 21: 210518_Stats_2400.png
210518_Stats_2400.png
Attachment 22: 210518_FEEtemps_2400.png
210518_FEEtemps_2400.png
Attachment 23: 210518_bias_2400.png
210518_bias_2400.png
Attachment 24: 210518_Deadtimes_R14_462.png
210518_Deadtimes_R14_462.png
  324   Tue May 18 14:57:43 2021 CBTuesday 18th May 16:00 - 20:00
15:58 Took shift over from OH and PP

Stats OK - attach 1
Temps OK - attach 2
Bias  OK - attach 3

System-wide checks
Clock - all pass
ADC - all fail (as before)
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x510 : 	 16  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA - all pass

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     22      8      7      4      1      4      2      2      2      2      7   : 37320
aida02 :     12      9      4      2      4      2      3      3      3      3      6   : 36856
aida03 :     19      6      7      2      2      2      2      4      3      3      6   : 37036
aida04 :     18      9     10      6      1      4      1      4      3      3      6   : 37168
aida05 :     17     12      2      1      2      3      3      3      3      3      6   : 36836
aida06 :     19      8      7      7      2      4      2      4      3      3      6   : 37468
aida07 :     28      5      4      1      2      4      2      3      2      2      7   : 37752
aida08 :     17      5      4      2      1      2      2      3      3      3      6   : 36396
aida09 :     21     10      4      5      4      4      3      3      3      3      6   : 37252
aida10 :     15     13      4      2      2      2      3      2      2      4      6   : 37284
aida11 :     22     13      3      3      4      3      1      4      3      3      6   : 37072
aida12 :      2      8      1      3      1      3      2      3      3      3      6   : 36472
aida13 :     23      5      4      4      1      4      3      2      3      3      6   : 36484
aida14 :     19      7      4      3      1      3      2      4      2      4      6   : 38116
aida15 :     20     10      2      2      3      2      3      2      3      3      6   : 36288
aida16 :     29      6      1      2      0      3      1      4      3      3      6   : 36724

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 17 11:43:04 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

Analysed R14_342. Attach 4. Dead time <5%



17:00

Stats OK - attach 5
Temps OK - as before
Bias  increasing quite quickly - attach 6

System-wide checks

System-wide checks
Clock - all pass
ADC - all fail (as before)
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x512 : 	 18  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA - all pass

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     28      7      8      4      1      4      2      2      2      2      7   : 37352
aida02 :     24     11      6      4      3      2      3      2      3      3      6   : 36440
aida03 :     17     11      5      2      1      2      2      3      3      3      6   : 36460
aida04 :     25     13     10      6      2      4      2      3      3      3      6   : 37036
aida05 :     23      7      3      2      2      3      3      3      3      3      6   : 36868
aida06 :      1     12      3      8      2      4      3      3      3      3      6   : 37140
aida07 :     22     12      1      1      2      4      2      2      2      2      7   : 37224
aida08 :     21      8      7      2      1      3      3      2      3      3      6   : 36356
aida09 :     23      7      4      6      4      3      2      4      3      3      6   : 37396
aida10 :     27      7      2      2      2      2      3      3      2      4      6   : 37764
aida11 :     12      5      3      5      4      3      1      4      3      3      6   : 37032
aida12 :     23      9      4      1      2      3      2      3      3      3      6   : 36612
aida13 :     10     12      6      3      1      4      3      2      3      3      6   : 36488
aida14 :     23      7      3      3      1      3      2      3      2      4      6   : 37604
aida15 :     22      6      6      4      2      2      3      2      3      3      6   : 36328
aida16 :     27      7      3      1      0      3      1      4      3      3      6   : 36724



18:00

Stats OK - attach 7
Temps OK - as before
Bias - attach 8

System-wide checks - as before except
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x517 : 	 23  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Analysed R14_371. Attach 9. Dead time below 5%.



18:35 MBS DAQ crashed



18:40
Stats OK - attach 9
Temps OK - as before
Bias - attach 10

System-wide checks - as before except
	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x51a : 	 26  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

18:42 MBS DAQ restarted - new file being opened.


19:36
Stats OK - attach 11
Temps OK - attach 12
Bias - attach 13

System-wide checks
Clock OK
ADC all fail

	
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x51e : 	 30  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA OK

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     24     11      5      4      1      4      2      3      2      2      7   : 37832
aida02 :      0      5      6      3      4      2      3      2      3      3      6   : 36328
aida03 :     15      8      5      3      2      2      1      3      4      3      6   : 37292
aida04 :     14      8      7      5      3      4      2      3      3      3      6   : 36936
aida05 :     31      6      1      1      2      3      3      3      2      4      6   : 37852
aida06 :      9      8      3      8      2      4      3      3      3      3      6   : 37140
aida07 :     21      7      2      1      2      4      1      3      2      2      7   : 37452
aida08 :      8      4      5      3      1      3      2      3      3      3      6   : 36528
aida09 :     24      7      3      5      4      3      2      4      3      3      6   : 37352
aida10 :     19      9      7      1      2      2      3      2      2      4      6   : 37284
aida11 :     27      5      0      5      4      3      1      4      3      3      6   : 37044
aida12 :     31      9      2      1      2      3      2      3      3      3      6   : 36612
aida13 :     21     10      4      3      1      4      3      2      3      3      6   : 36484
aida14 :     22      6      4      2      1      3      2      4      2      4      6   : 38088
aida15 :     21      9      3      4      2      2      3      2      3      3      6   : 36300
aida16 :     23     11      4      2      1      4      2      3      3      3      6   : 36724


19:43 Shift handed over to LS
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Temps1.png
Temps1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Analysis_R14_342
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252065796 ( 1015168.3 Hz)
        Other data format:    9854206 (   39686.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        242 (       1.0 Hz)
                               RESUME:        241 (       1.0 Hz)
                              SYNC100:      32924 (     132.6 Hz)
                              WR48-63:      32924 (     132.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    9787875 (   39419.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     499592 (    2012.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      248.300 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.028               16.674
  1                4.476                0.000
  2                0.095                0.000
  3                4.784                0.000
  4                0.046               31.798
  5                8.905                0.000
  6                0.093                3.062
  7                8.121                0.000
  8                0.937                0.000
  9                0.000               59.989
 10                0.965                0.000
 11                0.025                0.000
 12                0.042                0.000
 13                0.000               63.078
 14                0.061                0.000
 15                0.217                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5702434       2900          0          0          3          3        703        703          0       1488          0      26833
  1   25200522       6341          0          0         31         30       3140       3140          0          0          0      55809
  2   10737118    2562866          0          0          3          3       1687       1687          0    2559486          0      44132
  3   32448674    2519917          0          0         37         37       4558       4558          0    2510727          0      63444
  4    3679860    1335111          0          0          5          5        577        577          0    1333947          0      11808
  5   37244520     759044          0          0         62         62       4940       4940          0     749040          0      34314
  6    9560033    1568630          0          0          3          3       1348       1348          0    1565928          0      23903
  7   40864979    1077667          0          0         66         66       5138       5138          0    1067259          0      40193
  8   17664728       4388          0          0          6          6       2188       2188          0          0          0      56741
  9    2360528        644          0          0          0          0        322        322          0          0          0       4285
 10   20213561       5020          0          0         12         12       2498       2498          0          0          0      34696
 11   11474187       2786          0          0          1          1       1392       1392          0          0          0      10298
 12    8797735       2188          0          0          1          1       1093       1093          0          0          0      49042
 13    2441760        604          0          0          0          0        302        302          0          0          0       3463
 14    7875814       1932          0          0          4          4        962        962          0          0          0      33004
 15   15799343       4168          0          0          8          8       2076       2076          0          0          0       7627
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   34.352s (  931.544 blocks/s,  58.222 Mb/s)
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Bias2.png
Bias2.png
Attachment 7: Stats3.png
Stats3.png
Attachment 8: Bias3.png
Bias3.png
Attachment 9: Analysis_R14_371
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  252182561 ( 1038396.9 Hz)
        Other data format:    9737439 (   40095.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        234 (       1.0 Hz)
                               RESUME:        234 (       1.0 Hz)
                              SYNC100:      32904 (     135.5 Hz)
                              WR48-63:      32904 (     135.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    9671163 (   39822.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     493534 (    2032.2 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      242.858 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.060               15.876
  1                5.312                0.000
  2                0.091                0.000
  3                3.590                0.000
  4                0.163               29.634
  5                9.407                0.000
  6                0.049                0.000
  7                6.820                0.000
  8                2.387                0.000
  9                0.000               29.724
 10                0.676                0.000
 11                0.043                0.000
 12                0.046                0.000
 13                0.000               57.696
 14                0.016                3.102
 15                0.214                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    5708701       2832          0          0          4          4        686        686          0       1452          0      26548
  1   24896226       6554          0          0         36         36       3241       3241          0          0          0      54251
  2   10734245    2512200          0          0          3          3       1683       1683          0    2508828          0      43373
  3   32202526    2481239          0          0         32         32       4391       4391          0    2472393          0      63125
  4    3649395    1322612          0          0         11         11        611        611          0    1321368          0      11610
  5   36874871     743171          0          0         64         64       4722       4722          0     733599          0      33565
  6    9551266    1557362          0          0          1          1       1434       1434          0    1554492          0      23558
  7   41439213    1089893          0          0         57         57       5374       5374          0    1079031          0      40478
  8   17460031       4238          0          0          7          7       2112       2112          0          0          0      55845
  9    2360192        582          0          0          0          0        291        291          0          0          0       4290
 10   20593873       5182          0          0          6          6       2585       2585          0          0          0      34440
 11   11697470       2942          0          0          2          2       1469       1469          0          0          0      10244
 12    8815559       2192          0          0          1          1       1095       1095          0          0          0      48502
 13    2454200        586          0          0          0          0        293        293          0          0          0       3481
 14    7942567       1924          0          0          2          2        960        960          0          0          0      32639
 15   15802226       3930          0          0          8          8       1957       1957          0          0          0       7585
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.879s (  944.541 blocks/s,  59.034 Mb/s)
Attachment 10: Stats4.png
Stats4.png
Attachment 11: Bias4.png
Bias4.png
  323   Tue May 18 06:50:53 2021 OH, PPTuesday 18th May 08:00 - 16:00
08:00 OH Takes over

      Statistics still look good from yesterday. Rate is still down.
      Dead time in FEEs 2 and 4 is 2.04%

      System wide checks clock ok
	
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x500 : 	 38  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      *Baseline reset*
      FPGA check ok
      Stats - attachment 1
      Temp - attachment 2
      Bias - attachment 3
      

08:11 With the decreased noise I am investigating whether we can lower the thresholds on DSSD2 y strips.
      I have set them to 0x19 on ASICS 1-3 and 0x64 on ASIC4
      The results of the test appear to indicate that it is the network link that we are saturating causing dead time
***	Before		                  After	
FEE	DT (s)  Dead time %	           DT (s)	Dead time %
0	0.013	0.004184451	           0.026	0.010025681
1	6.352	2.04458693	           4.097	1.57981599
2	0.295	0.09495484	           0.292	0.112596112
3	6.349	2.043621288	           8.255	3.183153771
4	0.021	0.006759497	           0.03	        0.011568094
5	0.285	0.091736032	           3.388	1.306423377
6	0.02	0.006437616	           0.149	0.057454865
7	1.219	0.392372712	           2.133	0.822491459
8	1.137	0.365978485	           0.457	0.176220627
9	0	0	                       0	0
10	1.633	0.525631369	           1.545	0.595756823
11	0.061	0.01963473	           0.166	0.064010118
12	0.059	0.018990968	           0.085	0.032776265
13	0	0	                       0	0
14	0.016	0.005150093	           0.023	0.008868872
15	0.282	0.09077039	           0.168	0.064781324


      Note that the dead time for FEE6 and 8 remains fairly low but the dead time for FEE4 has increased

09:00 They are stopping the run to decrease the bias on the bPlast detector
09:25 No noticable change in AIDA
      They will now begin to change the spill structure.

09:46 While they are optimising the 1s spill structure raised the thresholds of FEE6 and 8 back to 0x20. Will start from here when optimising the thresholds post spill change

10:21 Beam is back on AIDA

10:39 System wide checks. 
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x506 : 	 6  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

11:00 
      Stats - attachment 4
      Temp - attachment 5
      Bias - attachment 6

12:46 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50b : 	 11  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed

14:36 System wide checks
- ADC calibration check failed (this is anticipated because waveforms are off). 
- 1 Failed test in White Rabbit:
		 Base 		Current 	Difference
aida05 fault 	 0x500 : 	 0x50f : 	 15  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
- All other systems passed
Attachment 1: 210518_0757_Stats.png
210518_0757_Stats.png
Attachment 2: 210518_0758_Temp.png
210518_0758_Temp.png
Attachment 3: 210518_0759_Bias.png
210518_0759_Bias.png
Attachment 4: 210518_1059_Stats.png
210518_1059_Stats.png
Attachment 5: 210518_1100_Temp.png
210518_1100_Temp.png
Attachment 6: 210518_1101_Bias.png
210518_1101_Bias.png
  322   Mon May 17 23:17:16 2021 DSJTuesday 18 May 0.00-8.00 shift

0.00 - Taking over from Lewis 

00.30 - Rates look ok (fig1), Ran R14_147 through analyser, dead times all good (fig2)

01.00 - Rates look ok (Fig3), Ran R14_153 through analyser, dead times all good (fig4)

01.30 - Rates look ok (Fig5), Ran R14_158 through analyser, dead times all good (fig6)

02.00 - Rates look ok (Fig7), Ran R14_164 through analyser, dead times all good (fig8)

Leakage current ok (fig 9), Temps OK (fig10), UCESB looks ok (fig11)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16

         Base         Current     Difference
aida05 fault      0x4da :      0x4ee :      20  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     43     12      6      4      1      4      2      2      1      3      7   : 38444
aida02 :     21     12      4      4      2      3      3      3      3      3      6   : 36980
aida03 :     23      8      4      2      2      4      2      3      3      3      6   : 36764
aida04 :     34     12     10      5      1      4      1      4      3      3      6   : 37224
aida05 :     18      5      5      1      2      3      2      2      1      3      7   : 38112
aida06 :     40     12      5      7      2      4      2      4      3      3      6   : 37552
aida07 :     40     10      6      2      2      4      2      2      2      2      7   : 37392
aida08 :     19     11      3      3      1      4      3      3      3      3      6   : 36980
aida09 :     43      6      2      4      3      4      3      3      3      3      6   : 37180
aida10 :     32     14     10      3      4      1      2      4      1      3      7   : 39280
aida11 :     40      7      2      5      4      3      1      4      3      3      6   : 37144
aida12 :     37     14     13      2      2      3      2      4      3      3      6   : 37396
aida13 :     18      9      3      1      1      5      3      3      3      3      6   : 37024
aida14 :     36     13      6      4      2      3      2      3      1      3      7   : 38872
aida15 :     32     11      7      4      3      4      2      3      2      4      6   : 38024
aida16 :     26      9      4      1      0      3      1      4      3      3      6   : 36752

 

02.30 - rates ok (fig 12), Ran R14_170 through analyser, dead times all good (fig13)

03.00 - rates ok (fig 14), Ran R14_175 through analyser, dead times all good (fig15)

03.05 - No beam, problem with SIS, don't know how long will be off

03.14 - Beam is back

03.30 - rates ok (fig 16), Ran R14_180 through analyser, dead times all good (fig17)

04.00 - rates ok (fig 18), Ran R14_186 through analyser, dead times all good (fig19)

Leakage current ok (fig 20), Temps OK (fig21), UCESB looks ok (fig22)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16


         Base         Current     Difference
aida05 fault      0x4da :      0x4f2 :      24  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     26     11      5      6      1      4      2      2      1      3      7   : 38416
aida02 :     16      9      5      4      2      4      2      2      3      3      6   : 36312
aida03 :      9      7      4      4      2      4      2      3      3      3      6   : 36764
aida04 :     36     11      8      4      3      4      1      3      4      3      6   : 37800
aida05 :     20      4      3      2      2      3      2      2      1      3      7   : 38112
aida06 :     22     13     10      7      2      4      3      3      4      3      6   : 38336
aida07 :     28      7      7      2      2      4      2      2      2      2      7   : 37336
aida08 :      2      3      4      4      1      3      2      3      3      3      6   : 36512
aida09 :     40      7      1      4      4      3      3      3      3      3      6   : 37096
aida10 :     26      9     12      2      3      2      2      4      1      3      7   : 39280
aida11 :     30      7      1      5      4      3      1      3      2      4      6   : 37600
aida12 :     15     10      6      4      2      3      2      4      3      3      6   : 37228
aida13 :      1      3      2      1      2      4      3      2      3      3      6   : 36316
aida14 :     34     12      7      4      2      3      2      3      1      3      7   : 38872
aida15 :     18     11      5      3      4      4      2      3      2      4      6   : 37968
aida16 :     22      7      4      2      0      3      1      4      3      3      6   : 36752

04.30 - rates ok (fig23), Ran R14_192 through analyser, dead times all good (fig24)

05.00 - rates ok (fig25), Ran R14_198 through analyser, dead times all good (fig26)

05.30 - Rates ok (Fig27), Ran R14_203 through analyser, dead times all good (fig28)

06.00 - Rates ok (Fig29), Ran R14_203 through analyser, dead times all good (fig30)

Leakage current ok (fig 31), Temps OK (fig32), UCESB looks ok (fig33)

System Wide Checks:

Clock status test result: Passed 16, Failed 0

Calibration test result: Passed 0, Failed 16

         Base         Current     Difference
aida05 fault      0x4da :      0x4f9 :      31  
White Rabbit error counter test result: Passed 15, Failed 1

FPGA Timestamp error counter test result: Passed 16, Failed 0


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     16      9      3      7      1      5      1      3      1      3      7   : 38744
aida02 :     18      3      5      2      3      3      3      2      3      3      6   : 36400
aida03 :     21      9      4      2      2      4      2      4      3      3      6   : 37276
aida04 :     32     12      7      3      3      4      1      3      4      3      6   : 37744
aida05 :     12      8      3      2      2      4      2      2      1      3      7   : 38240
aida06 :     28     14      4      8      2      4      3      3      3      3      6   : 37280
aida07 :     40     12      5      2      2      4      1      3      2      2      7   : 37648
aida08 :     13      8      4      4      1      3      2      3      3      3      6   : 36596
aida09 :     35      5      2      4      3      4      2      4      3      3      6   : 37396
aida10 :      6     10      4      1      1      3      2      2      1      3      7   : 38024
aida11 :     41      9      1      4      4      3      2      4      3      3      6   : 37372
aida12 :     29     11      9      4      2      3      2      4      3      3      6   : 37340
aida13 :     24     10      5      1      2      4      3      2      3      3      6   : 36512
aida14 :     34     16      5      4      2      3      2      3      1      3      7   : 38872
aida15 :     42      8      6      2      4      4      2      3      2      4      6   : 38024
aida16 :      8      9      6      1      0      3      1      4      3      3      6   : 36712

06.30 - Rates look ok (Fig34), Ran R14_215 through analyser, dead times all good (fig35)

07.00 - Rates look ok (Fig36), Ran R14_221 through analyser, dead times all good (fig 37)

07.25 - Rates look ok (Fig38), Ran R14_226 through analyser, dead times all good (fig 39)

Attachment 1: Screenshot_2021-05-18_Statistics_aidas-gsi(1).png
Screenshot_2021-05-18_Statistics_aidas-gsi(1).png
Attachment 2: R14_147_deadtime_23.41.06.png
R14_147_deadtime_23.41.06.png
Attachment 3: Screenshot_2021-05-18_Statistics_aidas-gsi(2).png
Screenshot_2021-05-18_Statistics_aidas-gsi(2).png
Attachment 4: R14_153_deadtime_00.07.12.png
R14_153_deadtime_00.07.12.png
Attachment 5: Screenshot_2021-05-18_Statistics_aidas-gsi(3).png
Screenshot_2021-05-18_Statistics_aidas-gsi(3).png
Attachment 6: R14_158_deadtime_00.32.42.png
R14_158_deadtime_00.32.42.png
Attachment 7: Screenshot_2021-05-18_Statistics_aidas-gsi(4).png
Screenshot_2021-05-18_Statistics_aidas-gsi(4).png
Attachment 8: R14_164_deadtime_01.02.32.png
R14_164_deadtime_01.02.32.png
Attachment 9: TMP.png
TMP.png
Attachment 10: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi.png
Attachment 11: Screenshot_2021-05-18_ucesb.png
Screenshot_2021-05-18_ucesb.png
Attachment 12: Screenshot_2021-05-18_Statistics_aidas-gsi(5).png
Screenshot_2021-05-18_Statistics_aidas-gsi(5).png
Attachment 13: R14_170_deadtime_01.35.24.png
R14_170_deadtime_01.35.24.png
Attachment 14: Screenshot_2021-05-18_Statistics_aidas-gsi(6).png
Screenshot_2021-05-18_Statistics_aidas-gsi(6).png
Attachment 15: R14_175_deadtime_01.59.37.png
R14_175_deadtime_01.59.37.png
Attachment 16: Screenshot_2021-05-18_Statistics_aidas-gsi(7).png
Screenshot_2021-05-18_Statistics_aidas-gsi(7).png
Attachment 17: R14_180_deadtime_02.31.32.png
R14_180_deadtime_02.31.32.png
Attachment 18: Screenshot_2021-05-18_Statistics_aidas-gsi(8).png
Screenshot_2021-05-18_Statistics_aidas-gsi(8).png
Attachment 19: R14_186_deadtime_03.00.27.png
R14_186_deadtime_03.00.27.png
Attachment 20: TMP.png
TMP.png
Attachment 21: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 22: Screenshot_2021-05-18_ucesb(1).png
Screenshot_2021-05-18_ucesb(1).png
Attachment 23: Screenshot_2021-05-18_Statistics_aidas-gsi(9).png
Screenshot_2021-05-18_Statistics_aidas-gsi(9).png
Attachment 24: R14_192_deadtime_03.31.26.png
R14_192_deadtime_03.31.26.png
Attachment 25: Screenshot_2021-05-18_Statistics_aidas-gsi(10).png
Screenshot_2021-05-18_Statistics_aidas-gsi(10).png
Attachment 26: R14_198_deadtime_04.00.21.png
R14_198_deadtime_04.00.21.png
Attachment 27: Screenshot_2021-05-18_Statistics_aidas-gsi(11).png
Screenshot_2021-05-18_Statistics_aidas-gsi(11).png
Attachment 28: R14_198_deadtime_04.27.46.png
R14_198_deadtime_04.27.46.png
Attachment 29: Screenshot_2021-05-18_Statistics_aidas-gsi(12).png
Screenshot_2021-05-18_Statistics_aidas-gsi(12).png
Attachment 30: R14_203_deadtime_05.png
R14_203_deadtime_05.png
Attachment 31: TMP.png
TMP.png
Attachment 32: Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-05-18_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 33: Screenshot_2021-05-18_ucesb(3).png
Screenshot_2021-05-18_ucesb(3).png
Attachment 34: Screenshot_2021-05-18_Statistics_aidas-gsi(13).png
Screenshot_2021-05-18_Statistics_aidas-gsi(13).png
Attachment 35: R14_215_deadtime_05.30.26.png
R14_215_deadtime_05.30.26.png
Attachment 36: Screenshot_2021-05-18_Statistics_aidas-gsi(14).png
Screenshot_2021-05-18_Statistics_aidas-gsi(14).png
Attachment 37: R14__221_DEADTIME_06.00.35.png
R14__221_DEADTIME_06.00.35.png
Attachment 38: Screenshot_2021-05-18_Statistics_aidas-gsi(15).png
Screenshot_2021-05-18_Statistics_aidas-gsi(15).png
Attachment 39: R14_226_deadtime_06.25.32.png
R14_226_deadtime_06.25.32.png
  321   Mon May 17 19:05:23 2021 LSMonday 17th May 20.00-24.00
20.30
Checked stats, seem okay (Fig 1)
Ran R14_99 through analyser, dead times all good (Fig 2)
Merger rate 1.8M items/s
Tape server 6.4MB/s

21.00
Checked stats seem okay (Fig 3)
Ran R14_106 through analyser, dead times all good (Fig 4)

21.30
Checked stats seem okay (Fig 5)
Ran R14_112 through analyser, dead times all good (Fig 6)

22.00
Stats okay (Fig 7)
FEE temps okay (Fig 8)
Rate spectra (Fig 9)
Leakage currents (Fig 10)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4e7 : 	 13  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.4M items/s
Tape server 6.3MB/s

Ran R14_119 through analyser, dead times all good (Fig 11)

22.30
Checked stats seem okay (Fig 12)
Ran R14_123 through analyser, dead times all good (Fig 13)

23.00
Checked stats seem okay (Fig 14)
Ran R14_129 through analyser, dead times all good (Fig 15)

23.30
Checked stats seem okay (Fig 16)
Ran R14_135 through analyser, dead times all good (Fig 17)

24.00
Stats okay (Fig 18)
FEE temps okay (Fig 19)
Leakage currents (Fig 20)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4eb : 	 17  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.5M items/s
Tape server 6.5MB/s

Ran R14_142 through analyser, dead times all good (Fig 21)
Attachment 1: 210517_Stats_2026.png
210517_Stats_2026.png
Attachment 2: 210517_Deadtime_R14_99.png
210517_Deadtime_R14_99.png
Attachment 3: 210517_Stats_2101.png
210517_Stats_2101.png
Attachment 4: 210517_Deadtimes_R14_106.png
210517_Deadtimes_R14_106.png
Attachment 5: 210517_Stats_2128.png
210517_Stats_2128.png
Attachment 6: 210517_Deadtimes_R14_112.png
210517_Deadtimes_R14_112.png
Attachment 7: 210517_Stats_2159.png
210517_Stats_2159.png
Attachment 8: 210517_FEEtemps_2159.png
210517_FEEtemps_2159.png
Attachment 9: 210517_Spectra_2159.png
210517_Spectra_2159.png
Attachment 10: 210517_Bias_2159.png
210517_Bias_2159.png
Attachment 11: 210517_Deadtimes_R14_119.png
210517_Deadtimes_R14_119.png
Attachment 12: 210517_Stats_2230.png
210517_Stats_2230.png
Attachment 13: 210517_Deadtimes_R14_123.png
210517_Deadtimes_R14_123.png
Attachment 14: 210517_Stats_2300.png
210517_Stats_2300.png
Attachment 15: 210517_Deadtimes_R14_129.png
210517_Deadtimes_R14_129.png
Attachment 16: 210517_Stats_2328.png
210517_Stats_2328.png
Attachment 17: 210517_Deadtimes_R14_135.png
210517_Deadtimes_R14_135.png
Attachment 18: 210517_Stats_2359.png
210517_Stats_2359.png
Attachment 19: 210517_FEEtemps_2359.png
210517_FEEtemps_2359.png
Attachment 20: 210517_bias_2359.png
210517_bias_2359.png
Attachment 21: 210517_deadtime_R14_142_1.png
210517_deadtime_R14_142_1.png
  320   Mon May 17 15:14:35 2021 LJWMonday 17th May 2021 16:00-20:00

16:25

Informed at shift handover that there has been no beam.

Statistics  - See attachment 1

Temperatures - See attachment 2

Bias & Leakage currents - See attachment 3

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit decoder status:

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4dd :      3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

18:00

Usecb scalars now showing DSSD implant rates.

Statistics  - See attachment 4

Temperatures - See attachment 5

Bias & Leakage currents - See attachment 6

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit decoder status

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx

 

19:45

Usecb scalars have continued to show DSSD implant rates since the last recorded elog check.

Statistics  - See attachment 7

Temperatures - See attachment 8

Bias & Leakage currents - See attachment 9

System Wide checks - All passed, EXCEPT:

ADC Calibration:

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White Rabbit decoder status:

     Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes

19:55 - Beam now back again - DSSD implants showing

 

 

Attachment 1: 170521_1.png
170521_1.png
Attachment 2: 170521_2.png
170521_2.png
Attachment 3: 170521_3.png
170521_3.png
Attachment 4: 170521_4.png
170521_4.png
Attachment 5: 170521_5.png
170521_5.png
Attachment 6: 170521_6.png
170521_6.png
Attachment 7: 170521_7.png
170521_7.png
Attachment 8: 170521_8.png
170521_8.png
Attachment 9: 170521_9.png
170521_9.png
  319   Fri May 14 19:01:09 2021 JSFriday 14th May 20:00-00:00
20:00 Taking over from CB

20:33 Doing full checks.
Temps ok elog:319/8  
Bias ok elog:319/1
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

FPGA Timestamp errors -
		 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Memory Info-
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     34     30      8      1      4      1      2      2      3      6   : 35772
aida02 :     37     30     28      5      3      1      2      4      2      3      6   : 36644
aida03 :     41     29     30     16      4      3      1      5      1      3      6   : 36588
aida04 :     13      6     23     12      4      4      4      4      2      2      6   : 35412
aida05 :     20     32     23     10      2      2      2      2      1      4      6   : 36736
aida06 :     14      5     10      5      1      3      2      3      2      3      6   : 35680
aida07 :     44     23     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     39     30     28      9      0      3      2      3      3      2      6   : 35308
aida09 :     41     31     28     10      2      5      1      3      1      3      6   : 35484
aida10 :     43     23     27      7      3      1      1      3      3      3      6   : 36916
aida11 :     39     35     17     10      3      3      2      2      2      3      6   : 35908
aida12 :     37     36     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     47     32     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     23     33     14     10      3      4      3      4      3      2      6   : 36164
aida15 :     44     28     24     12      3      2      4      3      3      2      6   : 35920
aida16 :     22     17     24     17      3      4      2      3      1      3      6   : 35648

Stats
Aida Correlation Info #8 - elog:319/2 same as elog:317/27 
Resume info #3 - elog:319/3 same as elog:317/26 
Pause info #2 - elog:319/4 same as elog:317/32 
AIDA disk info #6 - elog:319/5 same as elog:317/31 
AIDA ADC data items - elog:319/6 same as elog:317/30 
Good Events - elog:319/7 same as elog:317/29
[I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see 
clearly once uploaded the screenshots] 

Analysis R5_108 elog:319/9
Pause 149 Resume 150
Highest deadtime FEE10 3%

21:01
ucesb ok - Max 1700 Hz Implant - 1MHz Decay

21:35
Temps ok
Bias ok
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     39     34     29      8      1      4      1      2      2      3      6   : 35772
aida02 :     46     29     28      5      3      2      2      3      2      3      6   : 36288
aida03 :     39     25     29     16      4      2      2      5      1      3      6   : 36660
aida04 :     14      6     19     12      4      4      3      4      2      2      6   : 35096
aida05 :     20     30     24     10      2      2      2      2      1      4      6   : 36736
aida06 :     26      4     11      5      2      4      3      2      2      3      6   : 35672
aida07 :     46     22     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     28     32     28      9      0      2      2      4      3      2      6   : 35664
aida09 :     32     30     29     10      3      5      2      3      1      3      6   : 35776
aida10 :     39     23     25      7      3      1      1      3      3      3      6   : 36868
aida11 :     40     31     21     10      3      3      1      4      1      3      6   : 35688
aida12 :     17     31     12     10      2      2      3      4      3      2      6   : 35772
aida13 :     35     28     22      7      4      2      3      4      3      2      6   : 36012
aida14 :     30     33     14     10      3      4      3      4      3      2      6   : 36192
aida15 :     34     26     24     12      3      2      4      3      3      2      6   : 35864
aida16 :     38     24     27     17      3      4      2      3      1      3      6   : 35816

Stats
Good Events - elog:319/10
AIDA ADC data items - elog:319/11
AIDA disk info #6 - elog:319/12 
Pause info #2 - elog:319/13
Resume info #3 - elog:319/14 
Aida Correlation Info #8 - elog:319/15  - rates showed zero, checked again and normal elog:319/17

Analysis R5_128 elog:319/16
Pause 167 Resume 166
Highest deadtime FEE10 5%

22:00
Temps ok
Stats ok 
ucesb ok

22:35 
Temps ok
Stats ok 
ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up 
again elog:319/17

22:54 They are closing the file, we are on R5_155

23:00 
Attachment 1: bias1.png
bias1.png
Attachment 2: Screenshot_2021-05-14_Statistics_aidas-gsi(10).png
Screenshot_2021-05-14_Statistics_aidas-gsi(10).png
Attachment 3: Screenshot_2021-05-14_Statistics_aidas-gsi(9).png
Screenshot_2021-05-14_Statistics_aidas-gsi(9).png
Attachment 4: Screenshot_2021-05-14_Statistics_aidas-gsi(8).png
Screenshot_2021-05-14_Statistics_aidas-gsi(8).png
Attachment 5: Screenshot_2021-05-14_Statistics_aidas-gsi(7).png
Screenshot_2021-05-14_Statistics_aidas-gsi(7).png
Attachment 6: Screenshot_2021-05-14_Statistics_aidas-gsi(6).png
Screenshot_2021-05-14_Statistics_aidas-gsi(6).png
Attachment 7: Screenshot_2021-05-14_Statistics_aidas-gsi(5).png
Screenshot_2021-05-14_Statistics_aidas-gsi(5).png
Attachment 8: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: R5_108_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256861958 ( 1503007.9 Hz)
        Other data format:    5058042 (   29596.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        150 (       0.9 Hz)
                              SYNC100:      32636 (     191.0 Hz)
                              WR48-63:      32636 (     191.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4992471 (   29213.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     207149 (    1212.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      170.899 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                4.459               72.353
  2                1.217               86.117
  3                3.357               28.177
  4                0.000              160.970
  5                0.495               76.386
  6                0.216               93.617
  7                1.418               40.259
  8                3.253               44.571
  9                0.000              168.837
 10                5.250                9.058
 11                1.956               49.014
 12                0.182               82.483
 13                0.000              160.585
 14                0.143               88.740
 15                3.053               35.456
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8464368       3303          0          0          2          2       1138       1138          0       1023          0      11424
  1   16399642       4233          0          0         17         18       2099       2099          0          0          0      22505
  2   13045073    1576129          0          0          6          6       1793       1793          0    1572531          0      18179
  3   26563169    1558422          0          0         14         14       3487       3487          0    1551420          0      26353
  4    2213090     510296          0          0          0          0        328        328          0     509640          0       5100
  5   15498485     314714          0          0          4          4       1986       1986          0     310734          0      15610
  6   12553775     609520          0          0          7          7       1573       1573          0     606360          0      10360
  7   23384876     446841          0          0         10         10       3029       3029          0     440763          0      18175
  8   21832531       5542          0          0         15         15       2756       2756          0          0          0      22662
  9    1045474        278          0          0          0          0        139        139          0          0          0       1866
 10   42357744      10552          0          0         43         43       5233       5233          0          0          0      12639
 11   20606319       4998          0          0         11         11       2488       2488          0          0          0       4351
 12   14311836       3576          0          0          6          6       1782       1782          0          0          0      20348
 13    3016503        780          0          0          0          0        390        390          0          0          0       1508
 14   13047482       3248          0          0          4          4       1620       1620          0          0          0      12826
 15   22521591       5610          0          0         10         10       2795       2795          0          0          0       3243
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:74697.445s (    0.428 blocks/s,   0.027 Mb/s)
Attachment 10: Screenshot_2021-05-14_Statistics_aidas-gsi(16).png
Screenshot_2021-05-14_Statistics_aidas-gsi(16).png
Attachment 11: Screenshot_2021-05-14_Statistics_aidas-gsi(15).png
Screenshot_2021-05-14_Statistics_aidas-gsi(15).png
Attachment 12: Screenshot_2021-05-14_Statistics_aidas-gsi(14).png
Screenshot_2021-05-14_Statistics_aidas-gsi(14).png
Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi(13).png
Screenshot_2021-05-14_Statistics_aidas-gsi(13).png
Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(12).png
Screenshot_2021-05-14_Statistics_aidas-gsi(12).png
Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(11).png
Screenshot_2021-05-14_Statistics_aidas-gsi(11).png
Attachment 16: R5_128_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256841478 ( 1482693.9 Hz)
        Other data format:    5078522 (   29317.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        167 (       1.0 Hz)
                               RESUME:        166 (       1.0 Hz)
                              SYNC100:      32644 (     188.4 Hz)
                              WR48-63:      32644 (     188.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5012901 (   28938.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     205165 (    1184.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      173.226 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.621               76.080
  2                1.556               81.876
  3                3.719               27.628
  4                0.000              162.954
  5                0.710               80.266
  6                0.170               93.769
  7                2.714               44.740
  8                3.483               45.565
  9                0.000              167.005
 10                8.522               10.948
 11                2.181               50.562
 12                0.202               85.035
 13                0.000              161.510
 14                0.146               88.959
 15                1.812               38.597
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8468806       3273          0          0          7          7       1112       1112          0       1035          0      11260
  1   16364560       4360          0          0         13         13       2167       2167          0          0          0      23505
  2   13265609    1594590          0          0          6          6       1881       1881          0    1590816          0      18254
  3   26054686    1576337          0          0         14         14       3422       3422          0    1569465          0      26152
  4    2223261     507850          0          0          0          0        362        362          0     507126          0       4952
  5   15152054     312213          0          0          8          7       1938       1938          0     308322          0      15336
  6   12659379     606961          0          0          4          4       1612       1612          0     603729          0      10099
  7   22669666     438118          0          0         14         14       2841       2841          0     432408          0      17594
  8   22337729       5564          0          0         15         15       2767       2767          0          0          0      22524
  9    1020352        240          0          0          0          0        120        120          0          0          0       1807
 10   42596491      10676          0          0         59         59       5279       5279          0          0          0      12076
 11   20672887       5060          0          0         13         13       2517       2517          0          0          0       4379
 12   14335521       3608          0          0          5          5       1799       1799          0          0          0      20105
 13    2812573        690          0          0          0          0        345        345          0          0          0       1405
 14   13325633       3260          0          0          4          4       1626       1626          0          0          0      12554
 15   22882271       5722          0          0          5          5       2856       2856          0          0          0       3163
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:78055.344s (    0.410 blocks/s,   0.026 Mb/s)
Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(17).png
Screenshot_2021-05-14_Statistics_aidas-gsi(17).png
Attachment 18: ucesb.png
ucesb.png
Attachment 19: _~.png
_~.png
Attachment 20: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 21: Screenshot_2021-05-14_Statistics_aidas-gsi(23).png
Screenshot_2021-05-14_Statistics_aidas-gsi(23).png
Attachment 22: Screenshot_2021-05-14_Statistics_aidas-gsi(22).png
Screenshot_2021-05-14_Statistics_aidas-gsi(22).png
Attachment 23: Screenshot_2021-05-14_Statistics_aidas-gsi(21).png
Screenshot_2021-05-14_Statistics_aidas-gsi(21).png
Attachment 24: Screenshot_2021-05-14_Statistics_aidas-gsi(20).png
Screenshot_2021-05-14_Statistics_aidas-gsi(20).png
Attachment 25: Screenshot_2021-05-14_Statistics_aidas-gsi(19).png
Screenshot_2021-05-14_Statistics_aidas-gsi(19).png
Attachment 26: Screenshot_2021-05-14_Statistics_aidas-gsi(18).png
Screenshot_2021-05-14_Statistics_aidas-gsi(18).png
Attachment 27: R5_157_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256650793 ( 1485751.4 Hz)
        Other data format:    5269207 (   30503.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        130 (       0.8 Hz)
                               RESUME:        129 (       0.7 Hz)
                              SYNC100:      32643 (     189.0 Hz)
                              WR48-63:      32643 (     189.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5203662 (   30124.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     223116 (    1291.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      172.741 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.094               74.418
  2                0.927               76.576
  3                3.655               26.911
  4                0.000              161.752
  5                0.023               80.653
  6                0.215               91.583
  7                1.871               44.002
  8                1.038               39.676
  9                0.000              166.718
 10                8.591               10.747
 11                3.917               45.280
 12                0.214               88.240
 13                0.000              163.408
 14                0.280               88.718
 15                1.416               44.122
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8604841       3308          0          0          0          0       1138       1138          0       1032          0      12424
  1   16359820       4228          0          0         13         13       2101       2101          0          0          0      24974
  2   13547296    1610585          0          0          6          6       1930       1930          0    1606713          0      19730
  3   25981004    1589167          0          0         14         13       3491       3491          0    1582158          0      28914
  4    2316430     549243          0          0          0          0        351        351          0     548541          0       5271
  5   15208801     341870          0          0          1          1       1974       1974          0     337920          0      16457
  6   12807457     657341          0          0          5          5       1667       1667          0     653997          0      10925
  7   22697274     479019          0          0         10         10       2849       2849          0     473301          0      19398
  8   22587880       5594          0          0          5          5       2792       2792          0          0          0      24704
  9    1078064        266          0          0          0          0        133        133          0          0          0       1963
 10   42543713      10740          0          0         43         43       5327       5327          0          0          0      13313
 11   20362862       5182          0          0         14         14       2577       2577          0          0          0       4606
 12   13994455       3436          0          0          7          7       1711       1711          0          0          0      21818
 13    2731986        666          0          0          0          0        333        333          0          0          0       1621
 14   13145005       3222          0          0          8          8       1603       1603          0          0          0      13539
 15   22683905       5340          0          0          4          4       2666       2666          0          0          0       3459
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:83137.008s (    0.385 blocks/s,   0.024 Mb/s)
  318   Fri May 14 17:30:56 2021 TDAnalysis of S496 files R5_18, R5_32 & R5_45
S496/R5_18 - attachment 1

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xa *except* aida09 0xf
                             n+n FEE64s 0x20
Sampling ADCs OFF


S496/R5_32 - attachment 2

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xa *except* aida09 & aida15 0xf
                             n+n FEE64s 0x20

Sampling ADCs ON, trigger threshold 15000 


S496/R5_45 - attachment 3

LEC fast discriminators - *all* channels disabled

Slow comparator thresholds - p+n FEE64s 0xc
                             n+n FEE64s 0x20

Sampling ADCs ON, trigger threshold 15000 


* N.B. Ignore 'elapsed idle time(s)' metric which does not work with WR timestamps
 
Attachment 1: R5_18
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260127700 ( 2662102.7 Hz)
        Other data format:    1792300 (   18342.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1843 (      18.9 Hz)
                               RESUME:       1843 (      18.9 Hz)
                              SYNC100:      32365 (     331.2 Hz)
                              WR48-63:      32365 (     331.2 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1723884 (   17641.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      65165 (     666.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       97.715 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1               36.653               41.693
  2               44.272               48.364
  3               46.942               50.228
  4                0.000               81.057
  5               27.042               29.539
  6               41.734               45.821
  7               43.168               46.980
  8               54.980               58.850
  9                4.477               17.487
 10               57.700               61.776
 11               43.911               47.867
 12               48.996               52.920
 13               21.829               26.853
 14               51.720               56.077
 15               53.092               56.077
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   16755449       4921          0          0        122        122       2151       2151          0        375          0       4078
  1   15729031       4184          0          0        103        103       1989       1989          0          0          0       7687
  2   17390463     502955          0          0        136        135       2294       2294          0     498096          0       5652
  3   14745088     476937          0          0        118        118       1907       1907          0     472887          0       8350
  4    2857057     295831          0          0          0          0        413        413          0     295005          0       2971
  5   19341027     128907          0          0        112        112       2426       2426          0     123831          0       5913
  6   17649809     199902          0          0        136        136       2168       2168          0     195294          0       3189
  7   17833639     143068          0          0        135        135       2201       2201          0     138396          0       5626
  8   17514114       4531          0          0        139        138       2127       2127          0          0          0       6021
  9   14922420       3686          0          0         23         23       1820       1820          0          0          0       1039
 10   16909910       4387          0          0        140        141       2053       2053          0          0          0       3196
 11   18661933       4896          0          0        151        151       2297       2297          0          0          0       1392
 12   17768788       4718          0          0        148        148       2211       2211          0          0          0       5274
 13   16872590       4258          0          0         96         96       2033       2033          0          0          0        605
 14   16747047       4335          0          0        134        135       2033       2033          0          0          0       3273
 15   18429335       4784          0          0        150        150       2242       2242          0          0          0        899
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:66231.297s (    0.483 blocks/s,   0.030 Mb/s)
Attachment 2: R5_32
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259165945 ( 2447522.6 Hz)
        Other data format:    2754055 (   26008.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1068 (      10.1 Hz)
                               RESUME:       1070 (      10.1 Hz)
                              SYNC100:      32395 (     305.9 Hz)
                              WR48-63:      32395 (     305.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2687127 (   25376.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     109002 (    1029.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      105.889 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.772               22.665
  2               28.336               32.781
  3                9.177               18.264
  4                0.000               94.253
  5                5.882               26.886
  6                8.897               16.463
  7               18.512               26.355
  8               29.893               33.327
  9                0.057               79.535
 10               45.010               46.522
 11               27.988               27.766
 12               32.050               37.161
 13                1.981               23.343
 14                0.002               96.477
 15               36.327               39.608
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   17728100       5332          0          0         32         32       2340       2340          0        588          0       6827
  1   15060064       3752          0          0         18         18       1858       1858          0          0          0      14360
  2   18674317     723732          0          0         99         99       2382       2382          0     718770          0       8046
  3   19118104     904863          0          0         39         39       2463       2463          0     899859          0      15890
  4    2260998     321658          0          0          0          0        326        326          0     321006          0       3138
  5   14506811     189277          0          0         25         25       1756       1756          0     185715          0       9208
  6   21798062     340177          0          0         43         43       2689       2689          0     334713          0       5553
  7   18561150     231264          0          0         70         70       2324       2324          0     226476          0       9337
  8   22060606       5804          0          0        136        136       2766       2766          0          0          0      10232
  9    5139884       1246          0          0          5          5        618        618          0          0          0       1276
 10   24215101       6255          0          0        190        191       2937       2937          0          0          0       4375
 11   22485598       5956          0          0        148        148       2830       2830          0          0          0       2034
 12   19527530       4883          0          0        108        109       2333       2333          0          0          0       8243
 13   14237341       3538          0          0          9          9       1760       1760          0          0          0        984
 14    2260329        600          0          0          1          1        299        299          0          0          0       8235
 15   21531950       5718          0          0        145        145       2714       2714          0          0          0       1264
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:65849.211s (    0.486 blocks/s,   0.030 Mb/s)
Attachment 3: R5_45
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257173968 ( 1605341.2 Hz)
        Other data format:    4746032 (   29625.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        149 (       0.9 Hz)
                              SYNC100:      32597 (     203.5 Hz)
                              WR48-63:      32597 (     203.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4680540 (   29217.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     198582 (    1239.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.199 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.826               60.560
  2                1.857               72.569
  3                3.101               18.747
  4                0.000              149.092
  5                0.725               60.979
  6                0.276               85.413
  7                1.668               27.289
  8                2.719               31.235
  9                0.000              157.899
 10                6.867               10.073
 11                2.634               42.733
 12                0.375               69.093
 13                0.000              148.656
 14                0.255               79.878
 15                3.499               32.266
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8318308       3120          0          0          3          3       1077       1077          0        960          0      11152
  1   17482671       4536          0          0         10         10       2258       2258          0          0          0      22118
  2   12591028    1472109          0          0          7          7       1790       1790          0    1468515          0      17237
  3   26467090    1465014          0          0         17         17       3574       3574          0    1457832          0      25917
  4    2005748     477188          0          0          0          0        331        331          0     476526          0       4838
  5   16244903     298514          0          0          5          5       2012       2012          0     294480          0      14514
  6   11816116     567974          0          0          7          7       1554       1554          0     564852          0       9683
  7   24437155     423701          0          0          8          8       3155       3155          0     417375          0      17498
  8   22741388       5724          0          0         10         10       2852       2852          0          0          0      21729
  9    1027419        274          0          0          0          0        137        137          0          0          0       1781
 10   39497864       9636          0          0         47         47       4771       4771          0          0          0      12114
 11   19835466       4892          0          0          8          8       2438       2438          0          0          0       4107
 12   15299334       3708          0          0          8          8       1846       1846          0          0          0      19448
 13    3576125        840          0          0          0          0        420        420          0          0          0       1437
 14   13120071       3182          0          0          7          7       1584       1584          0          0          0      12115
 15   22713282       5620          0          0         12         12       2798       2798          0          0          0       2894
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:63108.355s (    0.507 blocks/s,   0.032 Mb/s)
  317   Fri May 14 14:57:32 2021 CB + OH, TD, NHFriday 14th May 16:00-20:00
16:00 Shift taken over from CA

16:00 
Temperatures OK. Attach 1. 
Upon reloading aida11 first returns "No respose" for all values. 
Upon refreshing, aida11 responds but aida 9 returns "No response". 
Upon refreshing again, attach 1.

Bias OK. Attach 2.

Statistics OK. Attach 3-7

System-wide checks
Clock OK
ADC - all fail
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     32     28     25      7      1      4      1      3      2      3      6   : 36112
aida02 :     43     28     29      5      4      2      3      4      1      3      6   : 36092
aida03 :     39     26     28      5      4      3      2      5      1      3      6   : 36428
aida04 :      9      5     21      8      5      4      4      4      2      2      6   : 35292
aida05 :     46     32     26      7      2      2      2      2      2      3      6   : 35768
aida06 :      4      5     16      1      3      4      2      4      1      3      6   : 35352
aida07 :     47     21     25      5      4      3      1      4      1      3      6   : 35604
aida08 :     62     33     32      6      0      2      3      3      3      2      6   : 35520
aida09 :     33     25     28      7      3      5      2      4      1      3      6   : 36140
aida10 :     47     29     22      6      3      2      2      2      2      3      6   : 35716
aida11 :     39     30     29      6      2      3      1      3      2      3      6   : 36124
aida12 :     38     36     20      6      3      2      3      3      3      2      6   : 35448
aida13 :     42     29     24      3      4      2      3      3      3      2      6   : 35440
aida14 :     31     28     18      7      3      3      4      4      2      2      6   : 35228
aida15 :     48     25     30      5      3      2      4      4      2      2      6   : 35272
aida16 :     37     16     28      6      3      3      2      2      2      3      6   : 35796

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021



16:15 Control handed over to OH temporarily.
16:25 AIDA stopped. OH attempts to update Merger. Discriminators off.
16:29 AIDA restart begins.
16:33 AIDA restarted.
16:40 Dead time still very high ~50% in most FEEs.
16:50 Checking effect of threshold on dead time. 
16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This
*reduces* deadtime to <10%.
      Sampling ADCs were disabled (switched off) late Wednesday evening.
17:01 DAQ can be restarted.
17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back.
17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20



17:40 OH hands back control.
Temperature OK - attach 10
Bias OK - attach 11
Stats - attach 12-17

System-wide checks
Clock - all pass
ADC - all pass
WR
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     56     35     27      8      1      4      1      2      2      3      6   : 35816
aida02 :     43     31     27      5      3      1      3      3      2      3      6   : 36404
aida03 :     45     28     28     16      4      2      2      4      1      3      6   : 36180
aida04 :      2      2     18     12      4      3      3      4      2      2      6   : 34872
aida05 :     28     28     24     10      2      2      2      2      1      4      6   : 36752
aida06 :     28     11      9      5      2      3      3      3      1      3      6   : 35064
aida07 :     52     24     23      9      4      3      1      2      2      3      6   : 35744
aida08 :     21     26     27      8      0      2      3      3      3      2      6   : 35284
aida09 :     40     32     26     10      3      5      1      3      1      3      6   : 35520
aida10 :     46     23     27      7      3      1      1      3      3      3      6   : 36928
aida11 :     49     31     20     10      2      2      1      2      2      3      6   : 35516
aida12 :     39     35     17      9      2      2      3      4      3      2      6   : 35940
aida13 :     40     32     20      7      4      2      3      4      3      2      6   : 36032
aida14 :     33     33     16     10      3      4      3      4      3      2      6   : 36236
aida15 :     53     27     25     12      3      2      4      4      2      2      6   : 35452
aida16 :     35     22     32     15      3      4      3      3      1      3      6   : 36060

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

18:30
Temperatures - OK (attach 18)
Bias OK (attach 19)
Stats - attach 20-24

System-wide checks
Clock - all pass
ADC - all pass
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     31     31      8      1      4      1      2      2      3      6   : 35788
aida02 :     42     31     29      5      3      2      4      4      1      3      6   : 36304
aida03 :     29     30     29     16      4      3      2      4      1      3      6   : 36276
aida04 :     28      4     20     12      4      3      4      4      2      2      6   : 35280
aida05 :     25     28     23     10      2      2      2      2      1      4      6   : 36724
aida06 :      5      3     10      5      2      4      2      2      2      3      6   : 35308
aida07 :     38     26     22      9      4      3      1      2      2      3      6   : 35688
aida08 :     52     30     28      8      0      2      2      3      3      2      6   : 35200
aida09 :     22     26     21     10      3      4      1      4      1      3      6   : 35704
aida10 :     37     28     25      7      3      1      1      3      3      3      6   : 36900
aida11 :     53     30     20     10      2      3      2      2      2      3      6   : 35908
aida12 :     37     34     17     10      2      2      3      4      3      2      6   : 35956
aida13 :     47     34     16      7      4      2      3      4      3      2      6   : 36012
aida14 :     37     29     17     10      3      4      3      4      3      2      6   : 36236
aida15 :     44     28     25     12      3      2      4      4      2      2      6   : 35424
aida16 :     40     21     33     15      3      4      1      3      1      3      6   : 35576

18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5%


19:40 Temperatures OK
Bias OK - attach 26
Stats - attach 27-32

System-wide checks

Clock, ADC - all pass
WR

	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc4 : 	 12  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     33     30      8      1      4      1      2      2      3      6   : 35788
aida02 :     51     30     30      5      3      2      3      4      1      3      6   : 36092
aida03 :     52     31     28     16      4      2      2      5      1      3      6   : 36744
aida04 :      6      8     20     12      4      4      3      4      2      2      6   : 35096
aida05 :     30     31     22     10      2      2      2      2      1      4      6   : 36752
aida06 :     12      6     10      5      2      3      3      2      2      3      6   : 35488
aida07 :     53     24     21      9      4      3      1      3      2      3      6   : 36228
aida08 :     36     28     28      9      0      2      3      3      3      2      6   : 35408
aida09 :     27     28     25     10      3      4      2      3      1      3      6   : 35548
aida10 :     50     23     26      7      3      1      1      3      3      3      6   : 36928
aida11 :     42     28     22     10      3      3      2      2      2      3      6   : 35944
aida12 :     39     35     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     43     34     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     42     32     16     10      3      4      3      4      3      2      6   : 36264
aida15 :     36     30     25     12      3      2      4      3      3      2      6   : 35920
aida16 :     40     21     25     15      3      4      2      3      1      3      6   : 35704

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19:40 Analysed R5_86. Attached. Dead time as before.

19:57 Shift handed over to James S
Attachment 1: Temp1.png
Temp1.png
Attachment 2: Bias1.png
Bias1.png
Attachment 3: Stats1-1.png
Stats1-1.png
Attachment 4: Stats1-2.png
Stats1-2.png
Attachment 5: Stats1-3.png
Stats1-3.png
Attachment 6: Stats1-4.png
Stats1-4.png
Attachment 7: Stats1-5.png
Stats1-5.png
Attachment 8: Stats1-6.png
Stats1-6.png
Attachment 9: R5_45_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257173968 ( 1605341.2 Hz)
        Other data format:    4746032 (   29625.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        149 (       0.9 Hz)
                              SYNC100:      32597 (     203.5 Hz)
                              WR48-63:      32597 (     203.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4680540 (   29217.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     198582 (    1239.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.199 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.826               60.560
  2                1.857               72.569
  3                3.101               18.747
  4                0.000              149.092
  5                0.725               60.979
  6                0.276               85.413
  7                1.668               27.289
  8                2.719               31.235
  9                0.000              157.899
 10                6.867               10.073
 11                2.634               42.733
 12                0.375               69.093
 13                0.000              148.656
 14                0.255               79.878
 15                3.499               32.266
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8318308       3120          0          0          3          3       1077       1077          0        960          0      11152
  1   17482671       4536          0          0         10         10       2258       2258          0          0          0      22118
  2   12591028    1472109          0          0          7          7       1790       1790          0    1468515          0      17237
  3   26467090    1465014          0          0         17         17       3574       3574          0    1457832          0      25917
  4    2005748     477188          0          0          0          0        331        331          0     476526          0       4838
  5   16244903     298514          0          0          5          5       2012       2012          0     294480          0      14514
  6   11816116     567974          0          0          7          7       1554       1554          0     564852          0       9683
  7   24437155     423701          0          0          8          8       3155       3155          0     417375          0      17498
  8   22741388       5724          0          0         10         10       2852       2852          0          0          0      21729
  9    1027419        274          0          0          0          0        137        137          0          0          0       1781
 10   39497864       9636          0          0         47         47       4771       4771          0          0          0      12114
 11   19835466       4892          0          0          8          8       2438       2438          0          0          0       4107
 12   15299334       3708          0          0          8          8       1846       1846          0          0          0      19448
 13    3576125        840          0          0          0          0        420        420          0          0          0       1437
 14   13120071       3182          0          0          7          7       1584       1584          0          0          0      12115
 15   22713282       5620          0          0         12         12       2798       2798          0          0          0       2894
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:63108.355s (    0.507 blocks/s,   0.032 Mb/s)
Attachment 10: Temps2.png
Temps2.png
Attachment 11: Bias2.png
Bias2.png
Attachment 12: Stats2-1.png
Stats2-1.png
Attachment 13: Stats2-2.png
Stats2-2.png
Attachment 14: Stats2-3.png
Stats2-3.png
Attachment 15: Stats2-4.png
Stats2-4.png
Attachment 16: Stats2-5.png
Stats2-5.png
Attachment 17: Stats2-6.png
Stats2-6.png
Attachment 18: Temp3.png
Temp3.png
Attachment 19: Bias3.png
Bias3.png
Attachment 20: Stats3-1.png
Stats3-1.png
Attachment 21: Stats3-2.png
Stats3-2.png
Attachment 22: Stats3-3.png
Stats3-3.png
Attachment 23: Stats3-4.png
Stats3-4.png
Attachment 24: Stats3-5.png
Stats3-5.png
Attachment 25: R5_67_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255938817 ( 1305497.2 Hz)
        Other data format:    5981183 (   30508.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        114 (       0.6 Hz)
                               RESUME:        113 (       0.6 Hz)
                              SYNC100:      32730 (     166.9 Hz)
                              WR48-63:      32730 (     166.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5915496 (   30173.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     257806 (    1315.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      196.047 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.111              102.399
  2                0.513              105.076
  3                3.587               39.451
  4                0.000              182.105
  5                0.792              103.780
  6                0.167              115.991
  7                1.244               72.680
  8                0.805               56.828
  9                0.000              190.127
 10                2.629                6.967
 11                0.295               76.751
 12                0.222              114.996
 13                0.000              188.225
 14                0.031              124.690
 15                2.491               63.947
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8223770       3364          0          0          2          2       1092       1092          0       1176          0      13825
  1   16628095       4338          0          0         12         12       2157       2157          0          0          0      28636
  2   13173045    1831289          0          0          3          3       1864       1864          0    1827555          0      22667
  3   27251797    1807380          0          0         13         13       3668       3668          0    1800018          0      33190
  4    2564061     622927          0          0          0          0        404        404          0     622119          0       6108
  5   14810679     385418          0          0          6          6       1903       1903          0     381600          0      19053
  6   12067625     742516          0          0          6          6       1616       1616          0     739272          0      12600
  7   21266971     549224          0          0         11         11       2723       2723          0     543756          0      22659
  8   23290358       5786          0          0          5          5       2888       2888          0          0          0      28199
  9    1157906        324          0          0          0          0        162        162          0          0          0       2443
 10   45608185      11489          0          0         27         26       5718       5718          0          0          0      16139
 11   20146427       4876          0          0          6          6       2432       2432          0          0          0       5579
 12   13411417       3300          0          0          7          7       1643       1643          0          0          0      24922
 13    2681820        654          0          0          0          0        327        327          0          0          0       1925
 14   11884997       2904          0          0          1          1       1451       1451          0          0          0      15892
 15   21771664       5394          0          0         15         15       2682       2682          0          0          0       3969
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:67311.180s (    0.475 blocks/s,   0.030 Mb/s)
Attachment 26: Stats_RESUME.png
Stats_RESUME.png
Attachment 27: Stats-CORRELATION.png
Stats-CORRELATION.png
Attachment 28: Bias4.png
Bias4.png
Attachment 29: Stats-good.png
Stats-good.png
Attachment 30: Stats-AIDAADC.png
Stats-AIDAADC.png
Attachment 31: Stats_DISC.png
Stats_DISC.png
Attachment 32: Stats_PAUSE.png
Stats_PAUSE.png
Attachment 33: R5_86_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256871525 ( 1549964.4 Hz)
        Other data format:    5048475 (   30462.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        133 (       0.8 Hz)
                               RESUME:        132 (       0.8 Hz)
                              SYNC100:      32617 (     196.8 Hz)
                              WR48-63:      32617 (     196.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4982976 (   30067.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     217412 (    1311.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      165.727 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.819               66.425
  2                0.365               78.785
  3                3.792               29.131
  4                0.000              149.995
  5                0.868               74.751
  6                0.080               85.583
  7                2.138               33.988
  8                2.899               39.410
  9                0.000              164.566
 10                6.499               11.006
 11                1.139               43.564
 12                0.944               80.601
 13                0.000              152.925
 14                0.102               87.597
 15                2.546               36.680
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8622288       3316          0          0          4          3       1158       1158          0        993          0      11812
  1   17129008       4510          0          0         12         12       2243       2243          0          0          0      24429
  2   13474006    1553973          0          0          6          6       1869       1869          0    1550223          0      19028
  3   26129234    1524704          0          0         20         20       3374       3374          0    1517916          0      27835
  4    2168475     521045          0          0          0          0        352        352          0     520341          0       5247
  5   15823578     326080          0          0          4          4       1963       1963          0     322146          0      16074
  6   12515224     622754          0          0          2          2       1625       1625          0     619500          0      10516
  7   23990039     458079          0          0         12         12       3099       3099          0     451857          0      18924
  8   22108276       5660          0          0         13         13       2817       2817          0          0          0      24206
  9    1062355        236          0          0          0          0        118        118          0          0          0       1949
 10   41223894      10244          0          0         40         40       5082       5082          0          0          0      13136
 11   20572413       4974          0          0          7          7       2480       2480          0          0          0       4577
 12   14036482       3446          0          0          3          3       1720       1720          0          0          0      21370
 13    3191210        818          0          0          0          0        409        409          0          0          0       1573
 14   12949121       3198          0          0          3          3       1596       1596          0          0          0      13450
 15   21875922       5438          0          0          7          7       2712       2712          0          0          0       3286
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:70812.570s (    0.452 blocks/s,   0.028 Mb/s)
  316   Fri May 14 07:05:08 2021 CAFriday 14th May 08:00 - 16:00 shift
08:00 CA takes over

08:28 Problem at UNILAC - no beam

09:30 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveforms)

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA timestamp errors:

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

09:36 FEE64 Temperatures ok - attachment 1

      Detector bias / leakage currents ok - attachment 2

      Statistics - attachments 3-8

09:50 beam is back - writing to file R4_265

09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA

10:10 FRS increase degrader thickness

10.28 all histograms, stats and merger stats zero'd

10.50 all system wide checks ok *except*
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      3      2      2      2      2      2      2      4      3      3      6   : 36860
aida02 :      7      3      5      1      2      5      2      4      3      3      6   : 37284
aida03 :      7      5      3      1      4      4      2      3      3      3      6   : 36756
aida04 :      3      1     20      4      2      3      3      5      1      3      6   : 36052
aida05 :     19     25     26      6      2      2      3      2      2      3      6   : 35828
aida06 :      1      3     16      1      3      4      1      5      1      3      6   : 35580
aida07 :      6      4      1      2      3      3      2      3      3      3      6   : 36552
aida08 :      8      5      2      3      2      3      2      2      2      4      6   : 37064
aida09 :      9      7      1      2      2      4      2      3      3      3      6   : 36652
aida10 :      2      5      3      1      1      4      2      3      3      3      6   : 36544
aida11 :     16      4      4      2      1      2      2      3      3      3      6   : 36384
aida12 :      2      1      1      3      1      2      2      3      3      3      6   : 36288
aida13 :      0      1      1      2      0      4      3      2      3      3      6   : 36184
aida14 :      6      3      2      0      2      3      2      3      3      3      6   : 36432
aida15 :     25      8      2      1      2      3      1      2      2      4      6   : 36836
aida16 :      3      5      1      2      2      2      3      2      3      3      6   : 36100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.50 DAQ continues S496/R4_304

11:18 FEE64 Temperatures ok - attachment 9

      Detector bias & leakage currents ok - attachment 10

      statistics - attachments 11-16

15:00 FEE64           avg. CPU usage (%)

      1               52
      2               55
      3               55
      4               60
      5               93
      6               50
      7               53
      8               55
      9               55
      10              51
      11              50
      12              67
      13              56
      14              55
      15              55
      16              58

15:08 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveform)

      WR decoder status:

      Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA errors:

 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

15:10 Temperatures ok - attachment 17

      Detector bias / leakage current ok - attachment 18

      Statistics - attachments 18 -24
Attachment 1: Screenshot_from_2021-05-14_08-32-16.png
Screenshot_from_2021-05-14_08-32-16.png
Attachment 2: Screenshot_from_2021-05-14_08-32-45.png
Screenshot_from_2021-05-14_08-32-45.png
Attachment 3: Screenshot_from_2021-05-14_08-33-11.png
Screenshot_from_2021-05-14_08-33-11.png
Attachment 4: Screenshot_from_2021-05-14_08-33-33.png
Screenshot_from_2021-05-14_08-33-33.png
Attachment 5: Screenshot_from_2021-05-14_08-34-12.png
Screenshot_from_2021-05-14_08-34-12.png
Attachment 6: Screenshot_from_2021-05-14_08-34-44.png
Screenshot_from_2021-05-14_08-34-44.png
Attachment 7: Screenshot_from_2021-05-14_08-35-17.png
Screenshot_from_2021-05-14_08-35-17.png
Attachment 8: Screenshot_from_2021-05-14_08-35-55.png
Screenshot_from_2021-05-14_08-35-55.png
Attachment 9: Screenshot_from_2021-05-14_10-17-25.png
Screenshot_from_2021-05-14_10-17-25.png
Attachment 10: Screenshot_from_2021-05-14_10-17-43.png
Screenshot_from_2021-05-14_10-17-43.png
Attachment 11: Screenshot_from_2021-05-14_10-22-39.png
Screenshot_from_2021-05-14_10-22-39.png
Attachment 12: Screenshot_from_2021-05-14_10-23-13.png
Screenshot_from_2021-05-14_10-23-13.png
Attachment 13: Screenshot_from_2021-05-14_10-23-41.png
Screenshot_from_2021-05-14_10-23-41.png
Attachment 14: Screenshot_from_2021-05-14_10-24-12.png
Screenshot_from_2021-05-14_10-24-12.png
Attachment 15: Screenshot_from_2021-05-14_10-24-37.png
Screenshot_from_2021-05-14_10-24-37.png
Attachment 16: Screenshot_from_2021-05-14_10-25-20.png
Screenshot_from_2021-05-14_10-25-20.png
Attachment 17: Screenshot_from_2021-05-14_14-07-48.png
Screenshot_from_2021-05-14_14-07-48.png
Attachment 18: Screenshot_from_2021-05-14_14-11-23.png
Screenshot_from_2021-05-14_14-11-23.png
Attachment 19: Screenshot_from_2021-05-14_14-13-21.png
Screenshot_from_2021-05-14_14-13-21.png
Attachment 20: Screenshot_from_2021-05-14_14-13-38.png
Screenshot_from_2021-05-14_14-13-38.png
Attachment 21: Screenshot_from_2021-05-14_14-14-12.png
Screenshot_from_2021-05-14_14-14-12.png
Attachment 22: Screenshot_from_2021-05-14_14-14-39.png
Screenshot_from_2021-05-14_14-14-39.png
Attachment 23: Screenshot_from_2021-05-14_14-15-06.png
Screenshot_from_2021-05-14_14-15-06.png
Attachment 24: Screenshot_from_2021-05-14_14-15-26.png
Screenshot_from_2021-05-14_14-15-26.png
  315   Thu May 13 23:58:05 2021 OHFriday 14th May 00:00 - 08:00
00:00 OH Takes over.
 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

OK
0x210 

 p+n Thresholds 0xa (100keV)
 n+n Threshold 0x20 (320keV)

01:00 System wide checks
 Clock status ok
 ADC calibration fail as expected (Waveforms off)
 WR Errors
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dbf : 	 7  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 FPGA Errors
	
                Base Current Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 Statistics ok - attachment 1
 Temp ok - attachment 2
 Bias and leakage current ok - attachment 3

01:42 Comparing the CS of AIDA3 and 4
      Stats have been running for a while (Believe since 5am)
      Looking at the counter ratio - attachment 4
      We still have a deadtime of around 14% in FEE4 (Assuming FEE3 has no dead time)

01:56 Noticed /media/ThirdDrive was at 97% full
      Let DESPEC locals know I would like to use the next run stop to change the directory.
      They said they could do it now so they stopped their run.
      Changed the symbolic link to /media/1e2....
      Now writing to R4_

03:26 MBS DAQ Stopped as beam stopped
03:33 Beam back now and DAQ running again

04:44 System wide checks as before
      Statistics ok - attachment 5
      Temperatures ok - attachment - 7
      Bias and leakage currents ok - attachment 6

05:53 MBS stops run as beam drops
06:00 MBS started a new run as beam is back

07:37 System wide checks clock ok and FPGA as before
      WR additional difference on FEE5
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc0 : 	 8  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics ok - attachment 8
     Temp ok - attachment 9
     Bias and leakage ok - attachment 10
 
Attachment 1: 210514_0100_Stats.png
210514_0100_Stats.png
Attachment 2: 210514_0101_Temp.png
210514_0101_Temp.png
Attachment 3: 210514_0102_Bias.png
210514_0102_Bias.png
Attachment 4: 210514_0141_CS_Stats.png
210514_0141_CS_Stats.png
Attachment 5: 210514_0442_Stats.png
210514_0442_Stats.png
Attachment 6: 210514_0443_Bias.png
210514_0443_Bias.png
Attachment 7: 210514_0443_Temp.png
210514_0443_Temp.png
Attachment 8: 210514_0737_Stats.png
210514_0737_Stats.png
Attachment 9: 210514_0738_Temp.png
210514_0738_Temp.png
Attachment 10: 210514_0739_Bias.png
210514_0739_Bias.png
Attachment 11: Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi(1).png
Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi.png
Screenshot_2021-05-14_Statistics_aidas-gsi.png
Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(1).png
Screenshot_2021-05-14_Statistics_aidas-gsi(1).png
Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(2).png
Screenshot_2021-05-14_Statistics_aidas-gsi(2).png
Attachment 16: Screenshot_2021-05-14_Statistics_aidas-gsi(3).png
Screenshot_2021-05-14_Statistics_aidas-gsi(3).png
Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(4).png
Screenshot_2021-05-14_Statistics_aidas-gsi(4).png
Attachment 18: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi.png
Attachment 19: Screenshot_2021-05-14_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-14_AIDA_Alerting_-_Grafana.png
Attachment 20: 600.png
600.png
Attachment 21: 601.png
601.png
  314   Thu May 13 21:21:25 2021 Philippos and Marcsystem wide checks

Wide checks at Time 22:22 CET

Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


         Base         Current     Difference
aida01 fault      0xf932 :      0xf933 :      1  
aida02 fault      0x62ec :      0x62ed :      1  
aida03 fault      0x8679 :      0x867a :      1  
aida04 fault      0xf0e4 :      0xf0e5 :      1  
aida05 fault      0x9db8 :      0x9dbc :      4  
aida06 fault      0x7f18 :      0x7f19 :      1  
aida07 fault      0xdd2c :      0xdd2d :      1  
aida08 fault      0x1557 :      0x1558 :      1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

             Base         Current         Difference
aida05 fault      0x0 :      0x2 :      2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     35     11      3      5      5      4      1      3      2      4      6   : 37876
aida02 :     20      5      4      3      2      5      2      4      3      3      6   : 37400
aida03 :     20      3      4      3      4      4      1      3      2      4      6   : 37640
aida04 :     25      6      2      3      4      4      3      3      3      3      6   : 37140
aida05 :     30     10      5      7      2      6      2      4      2      4      6   : 38776
aida06 :     21     11      4      4      3      5      2      4      3      3      6   : 37548
aida07 :     23      8      4      4      2      6      1      3      2      4      6   : 37852
aida08 :     27      7      2      3      4      3      2      4      3      3      6   : 37284
aida09 :     31      6      5      6      3      3      2      4      3      3      6   : 37372
aida10 :     22      5      3      4      3      6      2      2      2      4      6   : 37616
aida11 :     23      5      4      2      5      2      1      3      2      4      6   : 37444
aida12 :     20      7      4      7      3      5      2      3      3      3      6   : 37096
aida13 :     28      5      3      3      1      5      2      3      3      3      6   : 36840
aida14 :     38     12      2      4      2      4      1      4      3      3      6   : 37144
aida15 :     27      9      5      4      2      5      3      2      3      3      6   : 36740
aida16 :     21      6      4      4      1      4      2      3      3      3      6   : 36740

 

Attachment 1: 2021-05-13_ucesb_scalers_22h22mn.png
2021-05-13_ucesb_scalers_22h22mn.png
Attachment 2: 2021-05-13_Currents_22h22mn.png
2021-05-13_Currents_22h22mn.png
Attachment 3: 2021-05-13_temperatures_22h22.png
2021-05-13_temperatures_22h22.png
Attachment 4: 2021-05-13-22h22mn_Stat-good-events.png
2021-05-13-22h22mn_Stat-good-events.png
Attachment 5: 2021-05-13-22h22mn_ADC-data-items.png
2021-05-13-22h22mn_ADC-data-items.png
Attachment 6: 2021-05-13_22h22mn-Correlation-info.png
2021-05-13_22h22mn-Correlation-info.png
Attachment 7: 2021-05-13_22h22mn-Pause-info.png
2021-05-13_22h22mn-Pause-info.png
Attachment 8: 2021-05-13_22h22mn-Resume-Info.png
2021-05-13_22h22mn-Resume-Info.png
Attachment 9: 2021-05-13_merger_22h22.png
2021-05-13_merger_22h22.png
Attachment 10: 2021-05-13_22h22mn-LAY-ID1.png
2021-05-13_22h22mn-LAY-ID1.png
Attachment 11: 2021-05-13_22h22mn-LAY-ID2.png
2021-05-13_22h22mn-LAY-ID2.png
Attachment 12: 2021-05-13_22h22mn-LAY-ID3.png
2021-05-13_22h22mn-LAY-ID3.png
Attachment 13: 2021-05-13_22h22mn-LAY-ID4.png
2021-05-13_22h22mn-LAY-ID4.png
  313   Thu May 13 19:52:36 2021 Philippos and MarcConnection problem

At 20:15, We lost connection with AnyDesk.

Liliana restarted Anydesk on the AIDA DAQ PC. All seemed fine on AIDA DAQ but there was a more generic issue on the main DAQ, possibly the timesorter.

Helena restarted all after ~30mn and it's all working again now.

 

 

Attachment 1: 2021-05-13_ucesb_scalers.png
2021-05-13_ucesb_scalers.png
  312   Thu May 13 17:05:42 2021 Muneerah, MArcThu 13 May 04:00-00:00

19:08 System Check

Rates from ucesb below 1 kHz

Atachment 1 Stats

Atachment 2 Temperatures

Atachment 3 Voltages

System wide checks are all ok except

ADC calibration all failed  
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White rabbit  
         Base         Current     Difference
aida01 fault      0xf932 :      0xf933 :      1  
aida02 fault      0x62ec :      0x62ed :      1  
aida03 fault      0x8679 :      0x867a :      1  
aida04 fault      0xf0e4 :      0xf0e5 :      1  
aida05 fault      0x9db8 :      0x9dba :      2  
aida06 fault      0x7f18 :      0x7f19 :      1  
aida07 fault      0xdd2c :      0xdd2d :      1  
aida08 fault      0x1557 :      0x1558 :      1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA all passed

Memory information

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      6      2      2      1      2      4      3      4      2      3      7   : 40424
aida02 :      3      4      2      2      1      2      1      3      3      3      7   : 40140
aida03 :     10     10      0      3      2      4      2      4      2      3      7   : 40280
aida04 :      3      2      0      2      2      3      2      3      1      4      7   : 40540
aida05 :      4      9      2      3      2      4      3      3      1      4      7   : 41048
aida06 :      8      6      1      1      2      3      2      2      3      3      7   : 40064
aida07 :      1      4      0      1      1      3      1      3      3      3      7   : 40196
aida08 :      1      4      2      2      2      4      3      4      2      3      7   : 40452
aida09 :      8      5      2      2      1      2      1      3      3      3      7   : 40168
aida10 :     22      5      4      2      1      3      1      4      2      3      7   : 39872
aida11 :      2      1      2      1      2      2      3      3      2      3      7   : 39632
aida12 :      8      5      2      2      1      3      1      4      2      3      7   : 39784
aida13 :     18      8      2      1      0      2      3      3      2      3      7   : 39624
aida14 :     23     12      2      2      1      2      2      4      2      3      7   : 40028
aida15 :      3      5      3      2      2      3      1      4      2      3      7   : 39844
aida16 :      0      2      1      2      1      4      2      4      2      3      7   : 40096

 

 

 

Attachment 1: 130521_1802_Stats.png
130521_1802_Stats.png
Attachment 2: 130521_1802_Temp.png
130521_1802_Temp.png
Attachment 3: 130521_1804_Voltages.png
130521_1804_Voltages.png
  311   Thu May 13 07:10:17 2021 CA, TDMay 13th 08:00 - 16:00 shift
08:10 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

      Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9db9 : 	 1  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

08:16 attempt to recalibrate ADCs using FADC align and control - unsuccessful on all

08:18 FEE64 temps ok - attachment 1

      bias/leakage currents ok - attachment 2

      statistics - attachments 3-8

10:13 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9db9 : 	 1  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

10:16 FEE64 temps ok - attachment 9

      bias/leakage currents ok - attachment 10

      statistics - attachments 11-16

12:16 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dba : 	 2  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

12:16 FEE64 temps ok - attachment 17

      bias/leakage currents ok - attachment 18

      statistics - attachments 19-25

14:08 all system wide checks ok *except*

      all FEE64 modules fail ADC calibration

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dba : 	 2  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FEE64 temps ok - attachment 16

      bias/leakage currents ok - attachment 27

      statistics - attachments 28-35
Attachment 1: Screenshot_from_2021-05-13_07-17-47.png
Screenshot_from_2021-05-13_07-17-47.png
Attachment 2: Screenshot_from_2021-05-13_07-18-57.png
Screenshot_from_2021-05-13_07-18-57.png
Attachment 3: Screenshot_from_2021-05-13_07-19-42.png
Screenshot_from_2021-05-13_07-19-42.png
Attachment 4: Screenshot_from_2021-05-13_07-20-45.png
Screenshot_from_2021-05-13_07-20-45.png
Attachment 5: Screenshot_from_2021-05-13_07-23-01.png
Screenshot_from_2021-05-13_07-23-01.png
Attachment 6: Screenshot_from_2021-05-13_07-23-35.png
Screenshot_from_2021-05-13_07-23-35.png
Attachment 7: Screenshot_from_2021-05-13_07-24-02.png
Screenshot_from_2021-05-13_07-24-02.png
Attachment 8: Screenshot_from_2021-05-13_07-25-15.png
Screenshot_from_2021-05-13_07-25-15.png
Attachment 9: Screenshot_from_2021-05-13_09-17-57.png
Screenshot_from_2021-05-13_09-17-57.png
Attachment 10: Screenshot_from_2021-05-13_09-18-21.png
Screenshot_from_2021-05-13_09-18-21.png
Attachment 11: Screenshot_from_2021-05-13_09-18-58.png
Screenshot_from_2021-05-13_09-18-58.png
Attachment 12: Screenshot_from_2021-05-13_09-19-32.png
Screenshot_from_2021-05-13_09-19-32.png
Attachment 13: Screenshot_from_2021-05-13_09-20-23.png
Screenshot_from_2021-05-13_09-20-23.png
Attachment 14: Screenshot_from_2021-05-13_09-21-05.png
Screenshot_from_2021-05-13_09-21-05.png
Attachment 15: Screenshot_from_2021-05-13_09-21-53.png
Screenshot_from_2021-05-13_09-21-53.png
Attachment 16: Screenshot_from_2021-05-13_09-22-18.png
Screenshot_from_2021-05-13_09-22-18.png
Attachment 17: Screenshot_from_2021-05-13_11-19-16.png
Screenshot_from_2021-05-13_11-19-16.png
Attachment 18: Screenshot_from_2021-05-13_11-19-30.png
Screenshot_from_2021-05-13_11-19-30.png
Attachment 19: Screenshot_from_2021-05-13_11-20-09.png
Screenshot_from_2021-05-13_11-20-09.png
Attachment 20: Screenshot_from_2021-05-13_11-20-33.png
Screenshot_from_2021-05-13_11-20-33.png
Attachment 21: Screenshot_from_2021-05-13_11-20-59.png
Screenshot_from_2021-05-13_11-20-59.png
Attachment 22: Screenshot_from_2021-05-13_11-21-29.png
Screenshot_from_2021-05-13_11-21-29.png
Attachment 23: Screenshot_from_2021-05-13_11-21-53.png
Screenshot_from_2021-05-13_11-21-53.png
Attachment 24: Screenshot_from_2021-05-13_11-22-30.png
Screenshot_from_2021-05-13_11-22-30.png
Attachment 25: Screenshot_from_2021-05-13_11-24-36.png
Screenshot_from_2021-05-13_11-24-36.png
Attachment 26: Screenshot_from_2021-05-13_13-09-43.png
Screenshot_from_2021-05-13_13-09-43.png
Attachment 27: Screenshot_from_2021-05-13_13-10-03.png
Screenshot_from_2021-05-13_13-10-03.png
Attachment 28: Screenshot_from_2021-05-13_13-10-28.png
Screenshot_from_2021-05-13_13-10-28.png
Attachment 29: Screenshot_from_2021-05-13_13-10-52.png
Screenshot_from_2021-05-13_13-10-52.png
Attachment 30: Screenshot_from_2021-05-13_13-11-22.png
Screenshot_from_2021-05-13_13-11-22.png
Attachment 31: Screenshot_from_2021-05-13_13-11-51.png
Screenshot_from_2021-05-13_13-11-51.png
Attachment 32: Screenshot_from_2021-05-13_13-12-24.png
Screenshot_from_2021-05-13_13-12-24.png
Attachment 33: Screenshot_from_2021-05-13_13-13-34.png
Screenshot_from_2021-05-13_13-13-34.png
Attachment 34: Screenshot_from_2021-05-13_13-14-16.png
Screenshot_from_2021-05-13_13-14-16.png
  310   Thu May 13 00:51:40 2021 Muneerah and OHThu 13 May 00:00-08:00
01:50 Beam on
The leakage current and the voltages been monitered every half an hour and logged in the excel sheet, looks decreasing.
Tempetratures OK

Rates OK

Attachment 1 Voltages

Atachment 2 Rates

Atachment 3 Temperatures

Attachment 4 ucesb rates less than 1 kHz


System wide check are ok except:

Base Current Difference
aida05 fault  0x36ca :  0x36ce :  4 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Base Current Difference
aida13 fault 0xa :  0xe9 :  223 
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 0 5 3 0 1 4 3 3 3 3 6 : 36760
aida02 : 7 9 0 2 2 4 2 3 3 3 6 : 36644
aida03 : 1 3 2 1 2 1 2 3 3 3 6 : 36188
aida04 : 3 1 2 0 2 3 2 3 3 3 6 : 36404
aida05 : 20 9 3 2 2 3 2 2 3 3 6 : 36104
aida06 : 4 10 0 2 2 4 2 2 3 3 6 : 36128
aida07 : 3 1 2 1 3 3 2 3 3 3 6 : 36500
aida08 : 2 4 3 2 3 4 2 3 2 3 6 : 35672
aida09 : 1 6 3 3 1 4 2 3 2 3 6 : 35588
aida10 : 5 6 4 2 1 3 2 3 2 3 6 : 35460
aida11 : 23 8 2 2 2 3 3 4 2 3 6 : 36348
aida12 : 8 4 2 2 2 3 2 4 2 3 6 : 36000
aida13 : 2 5 3 1 2 4 3 3 2 3 6 : 35840
aida14 : 3 6 2 2 2 3 2 4 2 3 6 : 35996
aida15 : 2 4 3 3 0 4 1 3 3 3 6 : 36280
aida16 : 7 7 5 1 3 4 2 3 2 3 6 : 35716



<strong>05:00 OH Took over at 04:00</strong>

System wide checks ok except

   Base   Current  Difference
aida05 fault   0x36ca :   0x36d0 :   6 
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


    Base   Current   Difference
aida05 fault   0x0 :   0x1 :   1 
aida13 fault   0xa :   0xe9 :   223 
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



05:41 AIDA 05 has dropped out. Will restart the system

05:58 DAQ recovered from powercycle
 p+n strips at 0xa
 n+n strips at 0x20
 ADC control register at 0xFF
 All waveforms turned off in LED

 Statistics - attachment 5
 Correlation scaler in FEE 3 and 4 are mostly tracking 1:1

06:34 Local users have been unable to recover MBS
      And have been unable to contact local experts

07:22 Bias and leakage currents ok - attachment 6
 Stats attachment -7
Attachment 1: 130521_0155_Voltages.png
130521_0155_Voltages.png
Attachment 2: 130521_0157_Stats.png
130521_0157_Stats.png
Attachment 3: 130521_0158_Temp.png
130521_0158_Temp.png
Attachment 4: 130521_0204_ucesb.png
130521_0204_ucesb.png
Attachment 5: 210513_0600_Stats.png
210513_0600_Stats.png
Attachment 6: 210513_0722_bias.png
210513_0722_bias.png
Attachment 7: 210513_0722_Stats.png
210513_0722_Stats.png
  309   Wed May 12 18:53:38 2021 JSWed 12 20:00-00:00
19:51 Shift change

Stats ok
ucesb ok, Max implants ~1500 Hz 

20:30
Stats ok
ucesb ok, Max implants ~1000 Hz
current ok (maybe leveling out)

20:51 
The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340.

20:59 

Check
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 16, Failed 0
WR:
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36ce : 	 4  
White Rabbit error counter test result: Passed 15, Failed 1
FPGA time:
			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0xb6 : 	 172  
FPGA Timestamp error counter test result: Passed 15, Failed 1

--- both errors above same module as earlier shift
	 
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      3      2      0      1      3      3      2      3      3      6   : 36092
aida02 :      6      4      4      2      2      3      1      3      3      3      6   : 36280
aida03 :      6      3      2      0      0      3      3      2      3      3      6   : 36048
aida04 :     10      3      1      1      3      3      1      3      3      3      6   : 36272
aida05 :     19      8      2      0      2      3      1      3      3      3      6   : 36268
aida06 :      2      2      0      2      3      4      2      2      3      3      6   : 36120
aida07 :      6      7      3      0      3      4      3      2      3      3      6   : 36416
aida08 :      7      5      4      0      3      2      2      3      3      3      6   : 36420
aida09 :     21      6      3      2      2      4      2      3      2      3      6   : 35700
aida10 :     13      3      1      1      3      3      1      4      2      3      6   : 35772
aida11 :     25      6      4      2      1      4      2      4      2      3      6   : 36180
aida12 :      0      4      0      3      2      3      3      3      2      3      6   : 35712
aida13 :     18      5      2      3      2      3      2      4      2      3      6   : 36080
aida14 :      2      1      1      1      2      3      2      4      2      3      6   : 35904
aida15 :      0      1      1      3      0      4      2      2      3      3      6   : 35960
aida16 :      4      5      3      2      3      4      1      3      2      3      6   : 35432

ucesb ok elog:309/1
current ok elog:309/2
Stats ok elog:309/3
temps ok elog:309/4

21:36
ucesb ok
Stats ok 

22:01
ucesb ok 1500 Hz peak implant
Stats ok 
temps ok
bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values.

22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate).
Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time.
Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume.
Oscar is going to raise the threshold on one of the y FEEs as a test
Increased aida4 slow comparator threshold changed
There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate?
Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler

23:40 FRS magnet issue, possibly power, no beam

00:03 bias ok elog:309/5 has come down because no beam
handing over to muneerah
Attachment 1: ucesb.png
ucesb.png
Attachment 2: bias1.png
bias1.png
Attachment 3: Screenshot_2021-05-12_Statistics_aidas-gsi.png
Screenshot_2021-05-12_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-05-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 5: bias2.png
bias2.png
  308   Wed May 12 15:03:10 2021 CB, OHWed 12 16:00-20:00
16:00 Shift change

Statistics OK (attach 1)
Temps OK (attach 2)
Bias OK (attach 3)

System wide checks
Clock, ADC - OK

WR
		 Base 		Current 	Difference
aida05 fault 	 0x36ca : 	 0x36cb : 	 1  
White Rabbit error counter test result: Passed 15, Failed 1


FPGA

			 Base 		Current 		Difference
aida13 fault 	 0xa : 	 0x4b : 	 65  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      5      6      1      1      2      2      3      3      3      6   : 36388
aida02 :     21     10      2      2      2      3      2      2      3      3      6   : 36100
aida03 :     25      5      3      1      0      3      3      2      3      3      6   : 36188
aida04 :      5      2      0      3      2      3      2      3      3      3      6   : 36484
aida05 :     16      6      3      1      2      4      1      3      2      4      6   : 37440
aida06 :      2      2      2      2      2      3      1      3      3      3      6   : 36216
aida07 :     16      9      3      0      3      4      3      2      3      3      6   : 36472
aida08 :      0      7      1      1      2      2      3      3      2      3      6   : 35560
aida09 :     15      7      4      2      2      4      2      3      2      3      6   : 35700
aida10 :      3      7      3      4      1      3      1      4      1      4      6   : 36788
aida11 :      5      1      3      1      1      4      3      3      2      3      6   : 35756
aida12 :      5      5      0      3      2      3      2      4      2      3      6   : 35996
aida13 :      2      5      1      2      2      4      3      3      2      3      6   : 35840
aida14 :     19      5      3      1      1      2      1      3      3      3      6   : 36100
aida15 :     20      6      6      2      1      3      2      2      3      3      6   : 36064
aida16 :      7      9      4      1      3      4      2      3      2      3      6   : 35716

File size change
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Mon May 10 16:47:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


16:20 Beam is back. Currently on File R1_200
16:43 Beam is being optimised.

17:00 UCESB OK
Stats OK (attach 4)
Temps OK
Bias OK (attach 5)
System-wide checks as before

17:30 Beam rate is quite high (up to 2000 Hz peak on ucesb). Stats still OK

17:52 Started compressing R1_1*

18:00 UCESB ok (attach 5)
Stats OK (attach 6)
Temps OK
Bias OK (attach 7)
System wide checks as before

19:00 UCESB ok (as before)
Stats OK (attach 8)
Temps OK
Bias OK (attach 9)
Merger OK (attach 10)

19:32 UCESB ok
Stats OK (attach 11)
Temps OK (attach 12)
Bias OK (attach 13)
System wide checks as before

19:51 Shift handed over to James S
Attachment 1: Stat1.png
Stat1.png
Attachment 2: Temp1.png
Temp1.png
Attachment 3: Bias1.png
Bias1.png
Attachment 4: Stats2.png
Stats2.png
Attachment 5: Bias2.png
Bias2.png
Attachment 6: Stats3.png
Stats3.png
Attachment 7: Bias3.png
Bias3.png
Attachment 8: ucesb1.png
ucesb1.png
Attachment 9: Stats4.png
Stats4.png
Attachment 10: Bias4.png
Bias4.png
Attachment 11: Merger.png
Merger.png
Attachment 12: ucesb2.png
ucesb2.png
Attachment 13: Stats5.png
Stats5.png
Attachment 14: Temps2.png
Temps2.png
Attachment 15: Bias5.png
Bias5.png
  307   Wed May 12 06:59:55 2021 OH, RDPWed 12 May 08:00-16:00
08:00 While the rates appeared to have quietened around 7am they are back to their previous rate now.

      They still seem to be working on the setting. Implant rate in AIDA about 8 per spill. Thick degrader is still in place
      They have collected 25 minutes of implantation data. They are analysing this data to confirm the PID

08:38 The PID confirmation was apparently run number 4. During this run they were implanting Cd in AIDA DSSD2

09:09 They are now taking data implanting somewhere in the snout. I have turned off no storage.
      Unfortunately we are writing to NOTAPE R3. I was unable to change it as they were already taking data
      Data rate 16.3MB/s
      ucesb shows similar rates in SC42 and AIDA01. We may be implanting Te in AIDA then.

09:35 They stopped the DAQ as their is no beam. Switched the tapeserver over to R1 in directory S496. Setting to no storage

09:47 Beam back turned off no storage. On file R1_10

11:04 Now that we have implants on the detector we get a better look at the strips
      From the 1d pattern it is obvious that there is an adapter card misaligned in DSSD1- attachments 1 and 2
      This adaptor board corresponds to FEE11

12:17 Leakage currents screenshot 512.png

13:30 Beam off. Expected back at 15:00...

14:09 Making use of the beam off time NH and HA entered S4 to investigate FEE1 and FEE11.
      They noticed FEE11 was off and have realigned that. All channels are now observed in that FEE
      They couldn't see anything wrong with the connector for FEE1. We are still missing a block of channels there.
      Rates are as before. No better, no worse.

15:31 Leakage currents screenshot 513.png. They are continuing to rise following the intervention in S4. It could just be because the temperature is rising...
Attachment 1: 1dImplants.png
1dImplants.png
Attachment 2: 1dImplants2.png
1dImplants2.png
Attachment 3: 512.png
512.png
Attachment 4: 513.png
513.png
  306   Wed May 12 02:07:59 2021 MA, BAWed 12 May 00:00-08:00

03:12 System Check

attachment 1 : Current

attachment 2 : Temperature

attachment 3 : Rates

 

   
Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

   
Calibration test result: Passed 16, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   
         Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

   
             Base         Current         Difference
aida13 fault      0xa :      0xf :      5  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22      5      7      2      1      2      2      3      3      3      6   : 36464
aida02 :      9      8      3      3      1      4      1      2      4      3      6   : 36916
aida03 :      5      2      6      2      0      3      2      3      3      3      6   : 36420
aida04 :      6      5      2      3      4      4      2      3      3      3      6   : 36800
aida05 :     17      6      6      2      2      4      1      3      2      4      6   : 37524
aida06 :      7     12      3      4      3      3      1      3      3      3      6   : 36460
aida07 :     17     11      5      1      3      3      3      2      3      3      6   : 36428
aida08 :      3      5      0      1      4      2      2      4      2      3      6   : 35924
aida09 :     27      6      4      2      0      2      2      2      3      3      6   : 35868
aida10 :     16     11      8      0      2      2      1      3      2      4      6   : 37272
aida11 :     15      2      2      3      1      4      3      4      2      3      6   : 36364
aida12 :      1      6      4      3      1      3      2      4      2      3      6   : 35988
aida13 :     22     14     10      2      4      2      2      4      2      3      6   : 36264
aida14 :     26     10      4      3      2      1      1      3      3      3      6   : 36184
aida15 :     14      2      3      2      2      4      1      2      3      3      6   : 35896
aida16 :      7      5      4      0      2      4      2      3      2      3      6   : 35588

06:10 DSSD1 rate high!

Attached 4

Called OH and he woke up to fix it :)

It was a problem with one of the ASIC according to what he says:

One of the ASICs HEC was running crazy!. Forced the ASICs to check their settings which brings them to back into line see attachment 9. This done by around 06:35

 

07:03 System check

attachment 5 Spectrum rate

attachment 6 Voltages

attachment 7 Rates

attachment 8 Temperature

System wide clock are all ok except

White rabbit

 Base         Current     Difference
aida05 fault      0x36ca :      0x36cb :      1  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA

 Base         Current         Difference
aida13 fault      0xa :      0x14 :      10  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Attachment 1: 1205_0310.png
1205_0310.png
Attachment 2: 1205_0311.png
1205_0311.png
Attachment 3: 1205_0312.png
1205_0312.png
Attachment 4: 120521_0630ucesb.png
120521_0630ucesb.png
Attachment 5: 120521_0639_SpecRates.png
120521_0639_SpecRates.png
Attachment 6: 120521_0655_Voltages.png
120521_0655_Voltages.png
Attachment 7: 120521_0657_Rates.png
120521_0657_Rates.png
Attachment 8: 120521_0685_Temp.png
120521_0685_Temp.png
Attachment 9: 120521_0635_ASICs.png
120521_0635_ASICs.png
  305   Tue May 11 07:21:24 2021 OHTuesday 11 May
08:20 System wide checks all ok except
		 Base 		Current 	Difference
aida01 fault 	 0xc10c : 	 0xc10e : 	 2  
aida02 fault 	 0xbe2a : 	 0xbe2c : 	 2  
aida03 fault 	 0xf665 : 	 0xf667 : 	 2  
aida04 fault 	 0xeabd : 	 0xeabf : 	 2  
aida05 fault 	 0x36c8 : 	 0x36ca : 	 2  
aida06 fault 	 0x2ea2 : 	 0x2ea4 : 	 2  
aida07 fault 	 0x9186 : 	 0x9188 : 	 2  
aida08 fault 	 0x5443 : 	 0x5445 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

and

	
		 Base 		Current 	Difference
aida01 fault 	 0xc10c : 	 0xc10e : 	 2  
aida02 fault 	 0xbe2a : 	 0xbe2c : 	 2  
aida03 fault 	 0xf665 : 	 0xf667 : 	 2  
aida04 fault 	 0xeabd : 	 0xeabf : 	 2  
aida05 fault 	 0x36c8 : 	 0x36ca : 	 2  
aida06 fault 	 0x2ea2 : 	 0x2ea4 : 	 2  
aida07 fault 	 0x9186 : 	 0x9188 : 	 2  
aida08 fault 	 0x5443 : 	 0x5445 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    Statistics still high - attachment 1
    Temperatures ok - attachment 2
    Bias and leakage currents ok - attachment 3
Attachment 1: 210511_0823_Stats.png
210511_0823_Stats.png
Attachment 2: 210511_0824_Temp.png
210511_0824_Temp.png
Attachment 3: 210511_0825_Bias.png
210511_0825_Bias.png
  304   Mon May 10 11:45:15 2021 NHConfusing behaviour of rates
Testing the behaviour of powering down the fast ADCs in aida01 (the hot FEE)

Set local control 2 to 0xff in aida01
Wait... see temperature drpos by ~3 degrees (PJCS reports much larger drop in T9)
So far so normal

*but*

aida05 rates go very high when this waveform is disabled!!
Set local control to 0x00 again
They drop back...

Recalibrated FADCs and back to normal
  303   Mon May 10 08:49:37 2021 TDMonday 10 May
09.51 All system wide checks OK *except*


		 Base 		Current 	Difference
aida05 fault 	 0x8441 : 	 0x8442 : 	 1  
aida06 fault 	 0xc5c5 : 	 0xc5c6 : 	 1  
aida07 fault 	 0x6f07 : 	 0x6f08 : 	 1  
aida08 fault 	 0x4814 : 	 0x4815 : 	 1  
White Rabbit error counter test result: Passed 12, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      4      6      3      5      5      2      3      2      3      6   : 36004
aida02 :      4      1      3      6      3      4      3      3      2      3      6   : 36040
aida03 :     12      8      5      2      3      3      3      4      2      3      6   : 36416
aida04 :      1      2      5      4      1      4      3      4      2      3      6   : 36388
aida05 :     22      8      4      2      4      5      1      4      1      4      6   : 37272
aida06 :      5      4      0      6      3      6      3      3      2      3      6   : 36276
aida07 :     38     16      7      7      1      6      1      3      3      3      6   : 37032
aida08 :     16      4      2      3      1      4      1      3      3      3      6   : 36384
aida09 :      2      3      7      6      5      5      2      3      2      3      6   : 36112
aida10 :     42     15      4      6      2      6      2      4      2      3      6   : 36768
aida11 :      1      5     13      7      3      3      3      3      2      3      6   : 36124
aida12 :     23      9      3      4      7      3      2      3      2      3      6   : 35988
aida13 :      2      6      7      3      4      4      3      2      2      3      6   : 35592
aida14 :     17     11      3      4      3      5      2      3      2      3      6   : 35980
aida15 :      4      7      5      2      1      6      2      3      2      3      6   : 35864
aida16 :      4      1      3      3      4      2      3      4      2      3      6   : 36264


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 09 15:10:52 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

09.52 All histograms, stats and merger stats zero'd

      FEE64 temps OK - attachment 1
      ADC data, disc/WR 28-27/pause/correlation stats OK - attachments 2-6
      Rate spectra - attachment 7
       ASIC check load
      Rate spectra - attachment 8
      1.8.W spectra 20us FSR - attachments 9 & 10
      Grafana - most recent 7 days DSSSD bias and leakage current - attachment 11
      Merger stats/tape server/merger - attachment 12
      DSSSD bias and leakage current OK - attachment 13

10.08 1.8.W spectra 20us, 200us, 2ms & 20ms FSR - attachments 14-21
       we are not missing extrinsic noise features at longer time ranges cf. 20us FSR

12.18 Cannot connect to port 8015, MIDAS terminal says "Killed"
      Restart MIDAS 8015 
      Collect FPGA baseline to avoid TCL error
      FEEs unaffected and running as before

13.25 FEEs power cycled to see if any change in weird noise/rates situation
      No change 
      ASICs synchronised and back to normal

14.34 Start a pulser walkthrough in May21/R10
      Attenuation 5x, Rep Rate 25 Hz
      Amplitudes 10..9...1

14.45 Back to 2 Hz @ 2V Pulser, walkthrough complete. No Storage again.

 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0 (NB. actually it is 12 MeV)
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

OK
Attachment 1: Screenshot_2021-05-10_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-10_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-05-10_Statistics_aidas-gsi.png
Screenshot_2021-05-10_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-10_Statistics_aidas-gsi(1).png
Screenshot_2021-05-10_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-10_Statistics_aidas-gsi(2).png
Screenshot_2021-05-10_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-10_Statistics_aidas-gsi(3).png
Screenshot_2021-05-10_Statistics_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-10_Statistics_aidas-gsi(4).png
Screenshot_2021-05-10_Statistics_aidas-gsi(4).png
Attachment 7: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(2).png
Attachment 10: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(3).png
Attachment 11: Screenshot_2021-05-10_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-10_AIDA_Alerting_-_Grafana.png
Attachment 12: 510.png
510.png
Attachment 13: 511.png
511.png
Attachment 14: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(4).png
Attachment 15: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(11).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(11).png
Attachment 16: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(5).png
Attachment 17: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(10).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(10).png
Attachment 18: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(6).png
Attachment 19: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(9).png
Attachment 20: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(7).png
Attachment 21: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(8).png
  302   Sun May 9 19:29:23 2021 TDSunday 9 May
Attachment 1: Screenshot_2021-05-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-05-09_Statistics_aidas-gsi.png
Screenshot_2021-05-09_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-09_Statistics_aidas-gsi(1).png
Screenshot_2021-05-09_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-09_Statistics_aidas-gsi(2).png
Screenshot_2021-05-09_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-09_Statistics_aidas-gsi(3).png
Screenshot_2021-05-09_Statistics_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-09_System_wide_Checks_aidas-gsi.png
Screenshot_2021-05-09_System_wide_Checks_aidas-gsi.png
Attachment 7: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(2).png
Attachment 10: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(3).png
Attachment 11: Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-09_Spectrum_Browser_aidas-gsi(4).png
Attachment 12: Screenshot_2021-05-09_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-09_AIDA_Alerting_-_Grafana.png
Attachment 13: 503.png
503.png
Attachment 14: 504.png
504.png
  301   Sat May 8 09:05:04 2021 TDSaturday 8 May
09.54 FEE64 temperatures OK - attachment 1
      ADC data item stats OK - attachment 2
       p+n FEE64s 11x <100k, 1x <200k
      Rate spectra - attachments 3 & 4
       before and after ASIC check load
      1.8.L spectra - attachments 5 & 6
      1.8.W spectra 20us FSR - attachments 7 & 8
      Disc, WR-28-47, pause, resume info data items - attachments 9-12
       note large number pause/resume for aida13 - legacy numbers since last restart/zero?
       attachment 16 for pause stats since stats zero at 10.02 - looks OK
      Grafana DSSSD bias & leakage currents for most recent 48h OK - atatchment 13
      DSSSD bias & leakage currents - attachment 14
      Merger stats/merger/tape server messages - attachment 15

10.02 All histograms and stats zero'd
      
10.06 All system wide checks OK *except*

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 06 15:29:51 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.25 Pulser peak width aida01 97 ch FWHM, aida02 350 ch FWHM
       peak widths are increasing cf. earlier this week - see https://elog.ph.ed.ac.uk/DESPEC/291 , https://elog.ph.ed.ac.uk/DESPEC/298
Attachment 1: Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-05-08_Statistics_aidas-gsi.png
Screenshot_2021-05-08_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi.png
Attachment 4: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(1).png
Attachment 5: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(2).png
Attachment 6: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(3).png
Attachment 7: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(4).png
Attachment 8: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(5).png
Attachment 9: Screenshot_2021-05-08_Statistics_aidas-gsi(1).png
Screenshot_2021-05-08_Statistics_aidas-gsi(1).png
Attachment 10: Screenshot_2021-05-08_Statistics_aidas-gsi(2).png
Screenshot_2021-05-08_Statistics_aidas-gsi(2).png
Attachment 11: Screenshot_2021-05-08_Statistics_aidas-gsi(3).png
Screenshot_2021-05-08_Statistics_aidas-gsi(3).png
Attachment 12: Screenshot_2021-05-08_Statistics_aidas-gsi(4).png
Screenshot_2021-05-08_Statistics_aidas-gsi(4).png
Attachment 13: Screenshot_2021-05-08_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-08_AIDA_Alerting_-_Grafana.png
Attachment 14: 500.png
500.png
Attachment 15: 501.png
501.png
Attachment 16: Screenshot_2021-05-08_Statistics_aidas-gsi(5).png
Screenshot_2021-05-08_Statistics_aidas-gsi(5).png
Attachment 17: Screenshot_2021-05-08_Statistics_aidas-gsi(5).png
Screenshot_2021-05-08_Statistics_aidas-gsi(5).png
Attachment 18: Screenshot_2021-05-08_Statistics_aidas-gsi(6).png
Screenshot_2021-05-08_Statistics_aidas-gsi(6).png
Attachment 19: Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-08_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 20: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(6).png
Attachment 21: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(7).png
Attachment 22: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(8).png
Attachment 23: Screenshot_2021-05-08_Statistics_aidas-gsi(7).png
Screenshot_2021-05-08_Statistics_aidas-gsi(7).png
Attachment 24: Screenshot_2021-05-08_Statistics_aidas-gsi(8).png
Screenshot_2021-05-08_Statistics_aidas-gsi(8).png
Attachment 25: Screenshot_2021-05-08_Statistics_aidas-gsi(9).png
Screenshot_2021-05-08_Statistics_aidas-gsi(9).png
Attachment 26: Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-08_Spectrum_Browser_aidas-gsi(9).png
Attachment 27: Screenshot_2021-05-08_Statistics_aidas-gsi(10).png
Screenshot_2021-05-08_Statistics_aidas-gsi(10).png
  300   Fri May 7 19:47:54 2021 PJCSHowTo mitigate excessive temperature in an FEE64

After tests in the Daresbury T9 system.

Disabling the waveform ADCs in an FEE64 which is running with the FPGA over temperature will drop the FPGA temperature by 10 degrees.

To disable the ADCs open the Local Control browser window and set the ADC Control register , @2, to 0xFF

The easiest way to restart the waveform ADCs correctly is to rerun SETUP from the control window selecting just the FEE that is affected.

Alternatively STOP acquisition, set the ADC Control  register back to 0 and rerun the calibrate ADCs in the FADC Align and Control browser window.

 

  299   Fri May 7 13:11:22 2021 NHFriday 7th May
14:11 - Alpha has been running most of morning

Just saw rates in tape spike to 6 MB/s... 
Stop output to tape and look:
ASIC check & load... all rates except aida04 back to 0
Start output to tape
Back to ~300 KB/s

R6_49 will be affected by this.
Others seem OK

System wide check failures:

		 Base 		Current 	Difference
aida01 fault 	 0xb405 : 	 0xb406 : 	 1  
aida02 fault 	 0xefc7 : 	 0xefc8 : 	 1  
aida03 fault 	 0xdaab : 	 0xdaac : 	 1  
aida04 fault 	 0x8f7c : 	 0x8f7d : 	 1  
aida05 fault 	 0xb5bd : 	 0xb5be : 	 1  
aida06 fault 	 0xeff1 : 	 0xeff2 : 	 1  
aida07 fault 	 0x8f57 : 	 0x8f58 : 	 1  
aida08 fault 	 0xbef7 : 	 0xbef8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x200 : 	 512  
aida13 fault 	 0x0 : 	 0x61 : 	 97  
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


The FPGA errors seem to come on FEEs with no WR errors or clock errors (including Lock/PLL page)

Temps & Rates OK

Will now stop Alpha run and check noise situation

Run stopped at R6_49

Switching to nostorage (NOTAPE/R2)

Set ASICs to threshold 0xa, Rates OK
Rate to Tape/MBS ca 9 MB/s

No change to situation since yesterday... good sign

During MBS test aida13 rebooted... crash log from ttyUSB5 attached

It was later noticed connector half off in aida05, reseated and after some time got noise levels OK again...
cables on top very sensitive, indicative of issues inside maybe(?). We avoid touching AIDA for now :)

A power cycle of all FEEs was performed after aida13's reboot to make sure things are good.
All systems check OK
Attachment 1: elog_locks.png
elog_locks.png
Attachment 2: elog_temps.png
elog_temps.png
Attachment 3: elog_rates.png
elog_rates.png
Attachment 4: elog_rates1.png
elog_rates1.png
Attachment 5: aida13.txt
06/19:25:39|Data Acquisition Statistics counters now cleared^M
06/19:25:55|Clear Statistics (1)^M
06/19:25:55|------------[ cut here ]------------^M
07/15:23:58|kernel BUG at mm/slab.c:2974!^M
07/15:23:58|Oops: Exception in kernel mode, sig: 5 [#1]^M
07/15:23:58|PREEMPT Xilinx Virtex440^M
07/15:23:59|Modules linked in: aidamem xdriver xh_spidev_register^M
07/15:23:59|NIP: c009211c LR: c00920b0 CTR: 00000007^M
07/15:23:59|REGS: c0391bf0 TRAP: 0700   Not tainted  (2.6.31)^M
07/15:23:59|MSR: 00021000 <ME,CE>  CR: 24022048  XER: 00000000^M
07/15:23:59|TASK = c036e318[0] 'swapper' THREAD: c0390000^M
07/15:23:59|GPR00: 00000001 c0391ca0 c036e318 c680daf0 c694003c 0000000a c6940020 00000009 ^M
07/15:23:59|GPR08: 0000001b c680dae0 00000cf0 c680dae0 24008042 00005aa8 c03a0000 00000020 ^M
07/15:23:59|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200 ^M
07/15:23:59|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000006 c680e400 ^M
07/15:23:59|NIP [c009211c] cache_alloc_refill+0x130/0x608^M
07/15:23:59|LR [c00920b0] cache_alloc_refill+0xc4/0x608^M
07/15:23:59|Call Trace:^M
07/15:23:59|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)^M
07/15:23:59|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc^M
07/15:23:59|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8^M
07/15:23:59|[c0391d40] [c0042938] __send_signal+0x78/0x260^M
07/15:23:59|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c^M
07/15:24:00|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c^M
07/15:24:00|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30^M
07/15:24:00|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240^M
07/15:24:00|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c^M
07/15:24:00|[c0391e40] [c0040bf4] update_process_times+0x30/0x70^M
07/15:24:00|[c0391e60] [c005a000] tick_periodic+0x34/0xe8^M
07/15:24:00|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120^M
07/15:24:00|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c^M
07/15:24:00|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18^M
07/15:24:00|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc^M
07/15:24:00|[c0391fb0] [c000172c] rest_init+0x70/0x84^M
07/15:24:00|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac^M
07/15:24:00|[c0391ff0] [c0000204] skpinv+0x194/0x1d0^M
07/15:24:00|Instruction dump:^M
07/15:24:00|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010 ^M
07/15:24:00|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840 ^M
07/15:24:00|Kernel panic - not syncing: Fatal exception in interrupt^M
07/15:24:00|Call Trace:^M
07/15:24:00|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)^M
07/15:24:00|[c0391a80] [c00345bc] panic+0x94/0x168^M
07/15:24:01|[c0391ad0] [c000bd44] die+0x178/0x18c^M
07/15:24:01|[c0391af0] [c000c000] _exception+0x164/0x1b4^M
07/15:24:01|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c^M
07/15:24:01|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608^M
07/15:24:01|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc^M
07/15:24:01|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8^M
07/15:24:01|[c0391d40] [c0042938] __send_signal+0x78/0x260^M
07/15:24:01|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c^M
07/15:24:01|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c^M
07/15:24:01|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30^M
07/15:24:01|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240^M
07/15:24:01|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c^M
07/15:24:01|[c0391e40] [c0040bf4] update_process_times+0x30/0x70^M
07/15:24:01|[c0391e60] [c005a000] tick_periodic+0x34/0xe8^M
07/15:24:01|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120^M
07/15:24:01|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c^M
07/15:24:01|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18^M
07/15:24:01|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc^M
07/15:24:01|[c0391fb0] [c000172c] rest_init+0x70/0x84^M
07/15:24:01|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac^M
07/15:24:02|[c0391ff0] [c0000204] skpinv+0x194/0x1d0^M
07/15:24:02|Rebooting in 180 seconds..
07/15:27:02|^MISOL Version 1.00 Date 9th January 2017
  298   Thu May 6 16:59:20 2021 TD, OHWednesday 6 May
18.09 NH has installed jumpers LK2-4 for *all* adaptor PCBs

      per FEE64 rate spectra - attachments 1-4
       attachment 4 with BNC PB-5 pulser 10Hz
      ADC data items stats - attachment 5
      1.8.W spectra - attachments 6 & 7

      Observe significant reduction in DSSSD/ribbon cable boundary channel rates
       - status quo ante Tuesday 4 May https://elog.ph.ed.ac.uk/DESPEC/291 ?


      1.8.L spectra - attachments 8 & 9
       pulser peak width aida01 85 ch FWHM, aida2 295 ch FWHM

19:21 Alpha run started May21/R6
      Statistics dominated by aida04 with a rate of 6k
      System wide checks all ok except:
			 Base 		Current 		Difference
aida13 fault 	 0x0 : 	 0x17 : 	 23  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Writing to file at ~300kB/s
     Rate spectra for aida04 shows dominated by first asic - attachment 10
     1*L spectra for aida04 asic 1 - attachment 11

20.25 per FEE64 rate spectra - attachment 12
      handful of hot channels aida06, 09 & 10
      ADC data item stats - attachment 13
Attachment 1: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(4).png
Attachment 2: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(5).png
Attachment 3: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(6).png
Attachment 4: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(7).png
Attachment 5: Screenshot_2021-05-06_Statistics_aidas-gsi(1).png
Screenshot_2021-05-06_Statistics_aidas-gsi(1).png
Attachment 6: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(8).png
Attachment 7: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(9).png
Attachment 8: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png
Attachment 9: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png
Attachment 10: 210506_1924_aida04Rate.png
210506_1924_aida04Rate.png
Attachment 11: 210506_1927_1*L.png
210506_1927_1*L.png
Attachment 12: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(2).png
Attachment 13: Screenshot_2021-05-06_Statistics_aidas-gsi.png
Screenshot_2021-05-06_Statistics_aidas-gsi.png
  297   Wed May 5 20:10:16 2021 OHWednesday 5 May Alpha run
21:10 After all of the works in S4 today the noise in the system was considerably worse
      It was decided that before starting a new alpha run a powercycle would be performed.
      The powercycle has had no effect on the rates. They are still terrible.

      The difference between yesterday and today is:
      - Adaptor cards for FEE9 and 13 were re-aligned as they were previously misaligned
      - bPlast has been plugged in and biased

      All system wide checks ok
      Statistics threshold at 0xa - attachment 1
      Rate spectra (FEE5 does have events in all channels just noise is dominated by a few) - attachment 2
      Waveforms - attachments 3 and 4

      Pulser turned off and threshold raised to 0x64
      Statistics - attachment 5

21:19 Alpha run started to May21/R4*


08:51 DAQ continues OK - file May21/R4_23

      FEE64 temps OK - attachment 8
      ADC data items OK - attachment 9 
      1.8.W spectra - attachments 6 & 7
       no change cf. yesterday pm
      Grafana - most recent 12h DSSSD bias & leakage current - attachment 10
      DSSSD biases & leakage currents - attachment 11
      Tape Server/Merger - attachment 12

 09:44 R4 stopped
Attachment 1: 210504_2105_Stats.png
210504_2105_Stats.png
Attachment 2: 210504_2107_Layout1.png
210504_2107_Layout1.png
Attachment 3: 210504_2108_layout7.png
210504_2108_layout7.png
Attachment 4: 210504_2109_Layout8.png
210504_2109_Layout8.png
Attachment 5: 210504_2115_stats.png
210504_2115_stats.png
Attachment 6: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi.png
Attachment 7: Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-06_Spectrum_Browser_aidas-gsi(1).png
Attachment 8: Screenshot_2021-05-06_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-06_Temperature_and_status_scan_aidas-gsi.png
Attachment 9: Screenshot_2021-05-06_Statistics_aidas-gsi.png
Screenshot_2021-05-06_Statistics_aidas-gsi.png
Attachment 10: Screenshot_2021-05-06_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-06_AIDA_Alerting_-_Grafana.png
Attachment 11: 401.png
401.png
Attachment 12: 402.png
402.png
  296   Wed May 5 16:57:48 2021 TDTuesday 5 May
17.55 Grafana DSSSD bias & leakage current - attachment 1
       Note abrupt decrease of DSSSD# 2 leakage current c. 1.5uA at 17.37 
      DSSSD bias & leakage currents - attachment 2
Attachment 1: Screenshot_2021-05-05_AIDA_Alerting_-_Grafana(1).png
Screenshot_2021-05-05_AIDA_Alerting_-_Grafana(1).png
Attachment 2: 401.png
401.png
  295   Tue May 4 21:26:43 2021 OHAlpha run
22:26 Tape server set up to write to directory May21
      DAQ stopped
      Slow comparator set to 0x64 for all FEE64 and an ASIC check performed
      Pulser turned off
      DAQ started
      ~300k in merger - 38kB per second to file
      Statistics good (all FEE show 0 bar FEE4 at 3k) - attachment 1
      All system wide checks ok apart from FPGA check
      	
	        Base	Current  	Difference
aida13 fault 	 0x0 : 	 0x62 : 	 98  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Histograms zeroed.
     DSSD1 at 130V
     DSSD2 at 100V

08:47 Found DAQ running at c. 3M data items/s / 13MB/s
      Switched Tape Server to no storage mode
      
[npg@aidas-gsi ~]$ ls -lrth /TapeData/May21
total 113G
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_0
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_1
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_2
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_3
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_4
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_5
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_6
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_7
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_8
-rw-rw-r--. 1 npg npg  64K May  4 22:24 R1_9
-rw-rw-r--. 1 npg npg 1.3G May  5 06:15 R1_10
-rw-rw-r--. 1 npg npg 2.0G May  5 06:17 R1_11
-rw-rw-r--. 1 npg npg 2.0G May  5 06:20 R1_12
-rw-rw-r--. 1 npg npg 2.0G May  5 06:23 R1_13
-rw-rw-r--. 1 npg npg 2.0G May  5 06:25 R1_14
-rw-rw-r--. 1 npg npg 2.0G May  5 06:28 R1_15
-rw-rw-r--. 1 npg npg 2.0G May  5 06:30 R1_16
-rw-rw-r--. 1 npg npg 2.0G May  5 06:33 R1_17
-rw-rw-r--. 1 npg npg 2.0G May  5 06:36 R1_18
-rw-rw-r--. 1 npg npg 2.0G May  5 06:38 R1_19
-rw-rw-r--. 1 npg npg 2.0G May  5 06:41 R1_20
-rw-rw-r--. 1 npg npg 2.0G May  5 06:43 R1_21
-rw-rw-r--. 1 npg npg 2.0G May  5 06:46 R1_22
-rw-rw-r--. 1 npg npg 2.0G May  5 06:49 R1_23
-rw-rw-r--. 1 npg npg 2.0G May  5 06:51 R1_24
-rw-rw-r--. 1 npg npg 2.0G May  5 06:54 R1_25
-rw-rw-r--. 1 npg npg 2.0G May  5 06:57 R1_26
-rw-rw-r--. 1 npg npg 2.0G May  5 06:59 R1_27
-rw-rw-r--. 1 npg npg 2.0G May  5 07:02 R1_28
-rw-rw-r--. 1 npg npg 2.0G May  5 07:04 R1_29
-rw-rw-r--. 1 npg npg 2.0G May  5 07:07 R1_30
-rw-rw-r--. 1 npg npg 2.0G May  5 07:10 R1_31
-rw-rw-r--. 1 npg npg 2.0G May  5 07:12 R1_32
-rw-rw-r--. 1 npg npg 2.0G May  5 07:15 R1_33
-rw-rw-r--. 1 npg npg 2.0G May  5 07:18 R1_34
-rw-rw-r--. 1 npg npg 2.0G May  5 07:20 R1_35
-rw-rw-r--. 1 npg npg 2.0G May  5 07:23 R1_36
-rw-rw-r--. 1 npg npg 2.0G May  5 07:25 R1_37
-rw-rw-r--. 1 npg npg 2.0G May  5 07:28 R1_38
-rw-rw-r--. 1 npg npg 2.0G May  5 07:31 R1_39
-rw-rw-r--. 1 npg npg 2.0G May  5 07:33 R1_40
-rw-rw-r--. 1 npg npg 2.0G May  5 07:36 R1_41
-rw-rw-r--. 1 npg npg 2.0G May  5 07:39 R1_42
-rw-rw-r--. 1 npg npg 2.0G May  5 07:41 R1_43
-rw-rw-r--. 1 npg npg 2.0G May  5 07:44 R1_44
-rw-rw-r--. 1 npg npg 2.0G May  5 07:46 R1_45
-rw-rw-r--. 1 npg npg 2.0G May  5 07:49 R1_46
-rw-rw-r--. 1 npg npg 2.0G May  5 07:52 R1_47
-rw-rw-r--. 1 npg npg 2.0G May  5 07:54 R1_48
-rw-rw-r--. 1 npg npg 2.0G May  5 07:57 R1_49
-rw-rw-r--. 1 npg npg 2.0G May  5 08:00 R1_50
-rw-rw-r--. 1 npg npg 2.0G May  5 08:02 R1_51
-rw-rw-r--. 1 npg npg 2.0G May  5 08:05 R1_52
-rw-rw-r--. 1 npg npg 2.0G May  5 08:07 R1_53
-rw-rw-r--. 1 npg npg 2.0G May  5 08:10 R1_54
-rw-rw-r--. 1 npg npg 2.0G May  5 08:13 R1_55
-rw-rw-r--. 1 npg npg 2.0G May  5 08:15 R1_56
-rw-rw-r--. 1 npg npg 2.0G May  5 08:18 R1_57
-rw-rw-r--. 1 npg npg 2.0G May  5 08:20 R1_58
-rw-rw-r--. 1 npg npg 2.0G May  5 08:23 R1_59
-rw-rw-r--. 1 npg npg 2.0G May  5 08:26 R1_60
-rw-rw-r--. 1 npg npg 2.0G May  5 08:28 R1_61
-rw-rw-r--. 1 npg npg 2.0G May  5 08:31 R1_62
-rw-rw-r--. 1 npg npg 2.0G May  5 08:34 R1_63
-rw-rw-r--. 1 npg npg 2.0G May  5 08:36 R1_64
-rw-rw-r--. 1 npg npg 2.0G May  5 08:39 R1_65
-rw-rw-r--. 1 npg npg 2.0G May  5 08:41 R1_66
-rw-rw-r--. 1 npg npg 1.6G May  5 08:44 R1_67
[npg@aidas-gsi ~]$ 

      Rate increased c. 06.15 
       multiple ASICs in multiple FEE64s hot HEC channels
      ASIC check load restores rates to those at beginning of May21/R1
      Switched Tape Server to storage mode - file May21/R1_67

08.53 All system wide checks OK *except* 

			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
aida08 fault 	 0x0 : 	 0x1 : 	 1  
aida13 fault 	 0x62 : 	 0x9380 : 	 37662  
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Most recent merger errors

MERGE Data Link (2497): bad timestamp  7 3 0xd1fc813c 0x073033d6 0x00001083773033d6 0x167c1083773033d6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ce7ff9 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1dd806e 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ed7fc1 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1fd803c 0x07303ba6 0x0000108377303ba6 0x167c108377303ba6 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1cf8066 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1de8110 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1ee8086 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1fe7ff7 0x07304376 0x0000108377304376 0x167c108377304376 0x167c108377304b46
MERGE Data Link (2497): bad timestamp  7 3 0xd1c38041 0x0e60a6d6 0x000010832e60a6d6 0x167c10832e60a6d6 0x167c108377304b46

09.04 FEE64 temps OK - attachment 2
      ADC data item stats - attachment 3
      1.8.L spectra - attachments 4 & 5
      1.8.W spectra - attachments 6 & 7
      Grafana DSSSD bias & Leakage current most recent 24h - attachment 8
      DSSSD bias & leakage current - attacment 9

11:32 R1 stopped to realign boards and check situation in S4
Attachment 1: 210504_2228_Stats.png
210504_2228_Stats.png
Attachment 2: Screenshot_2021-05-05_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-05_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-05-05_Statistics_aidas-gsi.png
Screenshot_2021-05-05_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(3).png
Attachment 8: Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-05_Spectrum_Browser_aidas-gsi(4).png
Attachment 9: Screenshot_2021-05-05_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-05_AIDA_Alerting_-_Grafana.png
Attachment 10: 400.png
400.png
  293   Tue May 4 14:26:50 2021 NHFEE64 Adapter wiring diagram and layout
A figure showing the FEE64 numbers (a previous diagram had 2/6 and 4/8 on the wrong side of the DSSD) and the wiring of pulser/HV
Also the jumpers connected to the boards

As always diagram is looking with direction of beam

Edit: Figure as signals (HV/Pulser) come from the beam right not beam left
Otherwise OK
Attachment 1: AIDA_Adapter_Configuration(1).png
AIDA_Adapter_Configuration(1).png
  292   Tue May 4 14:05:48 2021 NHMACB Cables
During the installation of the 4 new HDMI cables for aidas13-16 the HDMI cables between the MACB Root and the MACB leaves were looped through the cable guide at the top to support them more
This may improve their connection and reduce the chance of WR errors seen in S460/S452
The connectors were all reseated a few times to clean the contacts too
Attachment 1: macb_front.jpg
macb_front.jpg
Attachment 2: macb_side.jpg
macb_side.jpg
  291   Tue May 4 13:53:22 2021 OH, TDTuesday 4 May
NH has re-checked alignment of AIDA adaptor PCBs cf. Elog:290

14.40 Rate spectra - attachment 1
      adc data items - attachment 2
       slow comparator 0xa
       stats for p+n much improved, n+n remain poor
      1.8.W spectra - attachments 3 & 4
      1.8.L spectra - attachments 5 & 6
       pulser peak widths aida01 69 ch FWHM, aida02 247 ch FWHM

17:31 Bias lowered to 100V on DSSD2 after noticing the leakage current has increased 2uA over the previous 3 hours. - attachment 7
Attachment 1: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-05-04_Statistics_aidas-gsi.png
Screenshot_2021-05-04_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-04_Spectrum_Browser_aidas-gsi(4).png
Attachment 7: 210504_Leakage.png
210504_Leakage.png
  290   Tue May 4 12:11:39 2021 OH TDBias test of triple
13:11 System wide checks all ok except *aida02 fails ADC calibration*
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Tue May 04 10:57:38 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:05 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

ASIC clocks have been synchronised and ADC re-calibrated

Rate spectra - attachment 1
Layout7 - attachment 2
Layout8 - attachment 3

Detectors at 130V

Statistics - attachment 4

V-I Curve - attachment 5
Voltage	Ch0	Ch1
10	1.345	1.49
20	2.01	2.285
30	2.405	2.8
40	2.635	3.13
50	2.82	3.37
60	2.95	3.57
70	3.08	3.725
80	3.185	3.865
90	3.26	3.98
100	3.33	4.105
110	3.42	4.22
120	3.525	4.35
130	3.64	4.46
140	3.79	4.37
150	3.98	4.37
150+10min	3.9	4.485
Attachment 1: 210504_1325_Rate.png
210504_1325_Rate.png
Attachment 2: 210504_1326_Layout7.png
210504_1326_Layout7.png
Attachment 3: 210504_1327_Layout08.png
210504_1327_Layout08.png
Attachment 4: 210504_1334_Stats.png
210504_1334_Stats.png
Attachment 5: VICurve.png
VICurve.png
  289   Sat May 1 08:45:19 2021 TDAIDA 24cm x 8cm DSSSD + triPlast assembly for S496
Photos provided by Nic Hubbard and Helena Albers from the assembly of 2x triPlast
plus 2x MSL type BB18(DS)-1000 24cm x 8cm detector assembly - attachments 1-3.

Spacers installed between MSL type BB18(DS)-1000 24cm x 8cm PCBs - attachment 4.

AIDA support stand in 24cm x 8cm DSSSD mode ready for installation of snout assembly - attachments 5 & 6

AIDA 24cm x 8cm DSSSD plus 2x TriPlast installed in S4 - view downstream towards upstream window 
of AIDA detctor assembly - attachment 7

AIDA 24cm x 8cm DSSSD plus 2x TriPlast snout assembly installed at S4 - attachment 8

Side view of AIDA + triPLast + FATIMA + DEGAS - attachment 9
Attachment 1: MicrosoftTeams-image.png
MicrosoftTeams-image.png
Attachment 2: IMG_7649.JPG
IMG_7649.JPG
Attachment 3: IMG_7650.JPG
IMG_7650.JPG
Attachment 4: MicrosoftTeams-image_(1).png
MicrosoftTeams-image_(1).png
Attachment 5: image-7becce0a-c5bb-431b-8af9-597e2b8d3829.jpg
image-7becce0a-c5bb-431b-8af9-597e2b8d3829.jpg
Attachment 6: image-3ba4cdee-a8f4-48e4-b087-03ddf07504e5.jpg
image-3ba4cdee-a8f4-48e4-b087-03ddf07504e5.jpg
Attachment 7: image-beb082d4-a965-453f-8804-394ce65a2f70.jpg
image-beb082d4-a965-453f-8804-394ce65a2f70.jpg
Attachment 8: IMG_7697.JPG
IMG_7697.JPG
Attachment 9: IMG_7708.JPG
IMG_7708.JPG
Attachment 10: IMG_7682.JPG
IMG_7682.JPG
Attachment 11: IMG_7681.JPG
IMG_7681.JPG
Attachment 12: FATIMA_Triple_AIDA.png
FATIMA_Triple_AIDA.png
  288   Fri Apr 30 08:38:12 2021 TDFriday 30 April
09.40 All system wide checks OK *except*

FEE64 module aida07 failed
Calibration test result: Passed 15, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Thu Apr 29 17:50:06 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:05 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

09.44 DSSSD bias & leakage currents Ok - attachment 1
      Merger time seq errors - attachment 2
       no merger errors since restart yesterday
      FEE64 temps OK - attachment 3
      good event stats - attachment 4
      Merger/Tape Server server consoles - attachment 5
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 6

      DAQ continues OK - no data storage
Attachment 1: 310.png
310.png
Attachment 2: 311.png
311.png
Attachment 3: Screenshot_2021-04-30_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-30_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2021-04-30_Statistics_aidas-gsi.png
Screenshot_2021-04-30_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-30_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-30_NewMerger_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-04-30_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-30_AIDA_Alerting_-_Grafana.png
  287   Thu Apr 29 16:56:42 2021 OHConfig changes for 16FEE
New set of ASIC settings produced 2021Apr29-13-16-00
- Note no longer odd/even for polarity of signal

New set of Options file for FEEs - Points to the new ASIC settings

New layouts for histograms grouped by side of the detector. Heavily weighted to the front side in terms of number of his

/MIDAS/Linux/startup/NewMerger Updated line from "./master64 -i 12 -l 16 -p 11001  &" to "./master64 -i 16 -l 16 -p 11001  &"

/MIDAS/DB/EXPERIMENTS/MERGE/Options/aidas-gsi/CONTENTS updated the number of links to 16.

System wide checks passed *except aida03 ADC calibration*

System left running to no storage.
  286   Wed Apr 28 17:39:35 2021 TDPreparations for setup of aida13-aida16
Updated /etc/dhcpd.conf

cd /etc
cp dhcpd.conf dhcpd.conf.BAK-280421

Add MAC addresses https://elog.ph.ed.ac.uk/DESPEC/285

aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15
aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b
aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed

New version of dhcpd.conf (and backup) - attachments 1 & 2

Cannot find dhcpd (and therefore re-read config and restart) in System - Administration -> Services
- has the dhcpd service been re-named or amalgamated with another service?



Updated /MIDAS/config/TclHttpd/aidas-gsi@8015/startup.tcl 

New version of startup.tcl (and backup) - attachments 3 & 4



Created rfs filesytems for aida13-aida24

[root@aidas-gsi]# cd /home/Embedded/XilinxLinux/ppc_4xx/rfs
[root@aidas-gsi rfs]# cd aida01
[root@aidas-gsi rfs]# tar cvf ../rfs.tar .

Create directories aida13-aida24, uid=10101, gid=npg, mode 777

[root@aidas-gsi rfs]# cd aida13; tar xvf ../rfs.tar .
 :
 :
etc

[root@aidas-gsi rfs]# ls -l 
total 1491880
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida01
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida02
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida03
drwxrwxrwx. 19 10101 npg        4096 Apr 24 03:51 aida04
drwxrwxrwx. 19 10101 npg        4096 Apr 24 03:51 aida05
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida06
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida07
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida08
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:35 aida09
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida10
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:34 aida11
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida12
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida13
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida14
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida15
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida16
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida17
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida18
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida19
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida20
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida21
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida22
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida23
drwxrwxrwx. 19 10101 npg        4096 Apr 28 16:32 aida24
-rw-r--r--.  1 root  root 1527582720 Apr 28 18:26 rfs.tar



Once dhcpd has been restarted it should be possible to power cycle all FEE64s, start system
and update firmware of newly installed FEE64s.
Attachment 1: dhcpd.conf
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
# Date of last update  Jan 12 2015 
#
authoritative;

ddns-update-style none; ddns-updates off;

# 2 days
#default-lease-time 172800;
# 4 days
default-lease-time 345600;
# 8 days
max-lease-time 691200;

option domain-search code 119 = string;

option domain-name "dl.ac.uk";
option domain-name-servers 193.62.115.16, 148.79.80.78;
option netbios-name-servers 148.79.160.89;
option netbios-node-type 8;
option nis-domain "nuclear.physics";
option nis-servers 193.62.115.77;


subnet 192.168.11.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.11.255;
   pool {
 range 192.168.11.118 192.168.11.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.11.99";
   next-server 192.168.11.99;

   host nnrpi1 {
     hardware ethernet b8:27:eb:bb:46:7b;
     fixed-address 192.168.11.251;
   }

   host nnrpi2 {
     hardware ethernet b8:27:eb:40:53:e8;
     fixed-address 192.168.11.117;
   }

   host aida01 {
     hardware ethernet d8:80:39:41:ba:8a;
     fixed-address 192.168.11.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01";
   }

   host aida02 {
     hardware ethernet d8:80:39:41:ba:22;
     fixed-address 192.168.11.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02";
   }

   host aida03 {
     hardware ethernet d8:80:39:41:d8:21;
     fixed-address 192.168.11.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03";
   }

   host aida04 {
     hardware ethernet d8:80:39:41:f6:b7;
     fixed-address 192.168.11.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04";
   }

   host aida05 {
     hardware ethernet d8:80:39:41:d7:cc;
     fixed-address 192.168.11.5;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   }

   host aida06 {
     hardware ethernet d8:80:39:41:ee:72;
     fixed-address 192.168.11.6;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   }

   host aida07 {
     hardware ethernet d8:80:39:41:b4:0c;
     fixed-address 192.168.11.7;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   }

   host aida08 {
     hardware ethernet d8:80:39:41:ba:2b;
     fixed-address 192.168.11.8;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   }

   host aida09 {
     hardware ethernet d8:80:39:41:f6:ee;
     fixed-address 192.168.11.9;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09";
   }

   host aida10 {
     hardware ethernet d8:80:39:41:ba:89;
     fixed-address 192.168.11.10;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10";
   }

   host aida11 {
     hardware ethernet d8:80:39:41:f6:5a;
     fixed-address 192.168.11.11;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11";
   }

   host aida12 {
     hardware ethernet d8:80:39:41:cf:ac;
     fixed-address 192.168.11.12;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12";
   }

   host aida13 {
     hardware ethernet d8:80:39:42:d:15;
     fixed-address 192.168.11.13;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida13";
   }

   host aida14 {
     hardware ethernet d8:80:39:42:d:b;
     fixed-address 192.168.11.14;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida14";
   }

   host aida15 {
     hardware ethernet d8:80:39:41:ee:10;
     fixed-address 192.168.11.15
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida15";
   }

   host aida16 {
     hardware ethernet d8:80:39:41:f6:ed;
     fixed-address 192.168.11.16;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida16";
   }
}


subnet 192.168.12.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.12.255;
   pool {
 range 192.168.12.100 192.168.12.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.12.99";
   next-server 192.168.12.99;

   host aida21 {
     hardware ethernet d8:80:39:41:d8:2a;
     fixed-address 192.168.12.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida21";
   }

   host aida22 {
     hardware ethernet 00:04:a3:2a:d0:26;
     fixed-address 192.168.12.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida22";
   }

   host aida23 {
     hardware ethernet 00:04:a3:2b:09:ce;
     fixed-address 192.168.12.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida23";
   }

   host aida24 {
     hardware ethernet 00:04:a3:2b:09:e8;
     fixed-address 192.168.12.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida24";
   }

}



Attachment 2: dhcpd.conf.BAK-280421
#
# DHCP Server Configuration file.
#   see /usr/share/doc/dhcp*/dhcpd.conf.sample  
#
# Date of last update  Jan 12 2015 
#
authoritative;

ddns-update-style none; ddns-updates off;

# 2 days
#default-lease-time 172800;
# 4 days
default-lease-time 345600;
# 8 days
max-lease-time 691200;

option domain-search code 119 = string;

option domain-name "dl.ac.uk";
option domain-name-servers 193.62.115.16, 148.79.80.78;
option netbios-name-servers 148.79.160.89;
option netbios-node-type 8;
option nis-domain "nuclear.physics";
option nis-servers 193.62.115.77;


subnet 192.168.11.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.11.255;
   pool {
 range 192.168.11.118 192.168.11.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.11.99";
   next-server 192.168.11.99;

   host nnrpi1 {
     hardware ethernet b8:27:eb:bb:46:7b;
     fixed-address 192.168.11.251;
   }

   host nnrpi2 {
     hardware ethernet b8:27:eb:40:53:e8;
     fixed-address 192.168.11.117;
   }

   host aida01 {
     hardware ethernet d8:80:39:41:ba:8a;
     fixed-address 192.168.11.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01";
   }

   host aida02 {
     hardware ethernet d8:80:39:41:ba:22;
     fixed-address 192.168.11.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02";
   }

   host aida03 {
     hardware ethernet d8:80:39:41:d8:21;
     fixed-address 192.168.11.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03";
   }

   host aida04 {
     hardware ethernet d8:80:39:41:f6:b7;
     fixed-address 192.168.11.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04";
   }

   host aida05 {
     hardware ethernet d8:80:39:41:d7:cc;
     fixed-address 192.168.11.5;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05";
   }

   host aida06 {
     hardware ethernet d8:80:39:41:ee:72;
     fixed-address 192.168.11.6;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06";
   }

   host aida07 {
     hardware ethernet d8:80:39:41:b4:0c;
     fixed-address 192.168.11.7;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07";
   }

   host aida08 {
     hardware ethernet d8:80:39:41:ba:2b;
     fixed-address 192.168.11.8;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08";
   }

   host aida09 {
     hardware ethernet d8:80:39:41:f6:ee;
     fixed-address 192.168.11.9;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09";
   }

   host aida10 {
     hardware ethernet d8:80:39:41:ba:89;
     fixed-address 192.168.11.10;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10";
   }

   host aida11 {
     hardware ethernet d8:80:39:41:f6:5a;
     fixed-address 192.168.11.11;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11";
   }

   host aida12 {
     hardware ethernet d8:80:39:41:cf:ac;
     fixed-address 192.168.11.12;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12";
   }
}


subnet 192.168.12.0 netmask 255.255.255.0 {
   option subnet-mask 255.255.255.0;
   option broadcast-address 192.168.12.255;
   pool {
 range 192.168.12.100 192.168.12.199;
   }
}

group {
   use-host-decl-names true;
   default-lease-time 3600;
   max-lease-time 14400;

   server-name "192.168.12.99";
   next-server 192.168.12.99;

   host aida21 {
     hardware ethernet d8:80:39:41:d8:2a;
     fixed-address 192.168.12.1;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida21";
   }

   host aida22 {
     hardware ethernet 00:04:a3:2a:d0:26;
     fixed-address 192.168.12.2;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida22";
   }

   host aida23 {
     hardware ethernet 00:04:a3:2b:09:ce;
     fixed-address 192.168.12.3;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida23";
   }

   host aida24 {
     hardware ethernet 00:04:a3:2b:09:e8;
     fixed-address 192.168.12.4;
    option root-path "/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida24";
   }

}



Attachment 3: startup.tcl

   Setting BGColor "BlanchedAlmond"
   Setting TextColor "red"

      Setting ProjectName  AIDA
      Setting ProjectImage AIDA.gif
      Setting ProjectURL   "http://npg.dl.ac.uk/NPG/Projects/Fair.html"


  Setting Option1Name STFC
  Setting Option1URL  "stfc.ac.uk"
  Setting Option1Image stfc.jpg


# global MessageLoggerServerHost
#  set  MessageLoggerServerHost nndhcp077.dl.ac.uk

#    package require  MessageLoggerClient 

#  Setting Option3Name "Elog Server"
#  Setting Option3URL  "http://nndhcp077.dl.ac.uk:8080/DataAcq/"

#    Install required services

    if {![namespace exists ::AIDA]} {namespace eval ::AIDA {puts "created namespace ::AIDA"}}
#    set ::AIDA::TestMode 1

    global SpectrumServerHost
    set SpectrumServerHost aidas

   package require XAIDAAccessClient  ;# for Client end - ie AIDA server workstation  sets MIDAS_XAIDAAccessClient_Exists (SOAP)

   package require XAD9252AccessClient ;# for Client end - ie AIDA server workstation  sets MIDAS_XAD9252AccessClient_Exists (SOAP)

namespace eval DataAcquisition {
     variable PROJECT AIDA
     variable ACQSERVERS
     set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12 aida13 aida14 aida15 aida16]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04]
     variable ACQSERVER [first $ACQSERVERS]
    variable DataTransferStreams; set DataTransferStreams 2
#     set ::DataAcquisition::EnableMerger 1
#     set ::DataAcquisition::EnableTape 1
    variable RDOGo 0 
}

namespace eval AIDA {
      variable ModuleMenu; set ModuleMenu $::DataAcquisition::ACQSERVERS
      variable TSMaster; set TSMaster aida01
      variable FEE64 [first $ModuleMenu]
      variable ConfigBase; set ConfigBase [file join $env(MIDASBASE) config TclHttpd]
}

    Setting DataBaseRoot "/MIDAS/DB"
    package require DataBaseAccessServer

    package require NetVarsServer 1.1
    package require SigTaskServer

#    package require MemSasServer

#    Install services for Run Control

namespace eval $::DataAcquisition::PROJECT {
    variable DBRootName [file join EXPERIMENTS AIDA]
    variable DBNodeBase 2012
}

  package require AIDARunControl
#  package require TapeServer
#  package require MergerControl
#  package require MERGERunControl


  Setting Option7Name "Project Home Page"
  Setting Option7URL  "http://npg.dl.ac.uk/NPG/Projects/Fair.html"

  Setting Option8Name "Server Software Home Page"
  Setting Option8URL  "http://npg.dl.ac.uk/MIDAS/download/WebServices.html"


Attachment 4: startup.tcl.BAK-280421

   Setting BGColor "BlanchedAlmond"
   Setting TextColor "red"

      Setting ProjectName  AIDA
      Setting ProjectImage AIDA.gif
      Setting ProjectURL   "http://npg.dl.ac.uk/NPG/Projects/Fair.html"


  Setting Option1Name STFC
  Setting Option1URL  "stfc.ac.uk"
  Setting Option1Image stfc.jpg


# global MessageLoggerServerHost
#  set  MessageLoggerServerHost nndhcp077.dl.ac.uk

#    package require  MessageLoggerClient 

#  Setting Option3Name "Elog Server"
#  Setting Option3URL  "http://nndhcp077.dl.ac.uk:8080/DataAcq/"

#    Install required services

    if {![namespace exists ::AIDA]} {namespace eval ::AIDA {puts "created namespace ::AIDA"}}
#    set ::AIDA::TestMode 1

    global SpectrumServerHost
    set SpectrumServerHost aidas

   package require XAIDAAccessClient  ;# for Client end - ie AIDA server workstation  sets MIDAS_XAIDAAccessClient_Exists (SOAP)

   package require XAD9252AccessClient ;# for Client end - ie AIDA server workstation  sets MIDAS_XAD9252AccessClient_Exists (SOAP)

namespace eval DataAcquisition {
     variable PROJECT AIDA
     variable ACQSERVERS
#     set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12 aida13 aida14 aida15 aida16]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04]
     set ACQSERVERS [list aida01 aida02 aida03 aida04 aida05 aida06 aida07 aida08 aida09 aida10 aida11 aida12]
#     set ACQSERVERS [list aida01 aida02 aida03 aida04]
     variable ACQSERVER [first $ACQSERVERS]
    variable DataTransferStreams; set DataTransferStreams 2
#     set ::DataAcquisition::EnableMerger 1
#     set ::DataAcquisition::EnableTape 1
    variable RDOGo 0 
}

namespace eval AIDA {
      variable ModuleMenu; set ModuleMenu $::DataAcquisition::ACQSERVERS
      variable TSMaster; set TSMaster aida01
      variable FEE64 [first $ModuleMenu]
      variable ConfigBase; set ConfigBase [file join $env(MIDASBASE) config TclHttpd]
}

    Setting DataBaseRoot "/MIDAS/DB"
    package require DataBaseAccessServer

    package require NetVarsServer 1.1
    package require SigTaskServer

#    package require MemSasServer

#    Install services for Run Control

namespace eval $::DataAcquisition::PROJECT {
    variable DBRootName [file join EXPERIMENTS AIDA]
    variable DBNodeBase 2012
}

  package require AIDARunControl
#  package require TapeServer
#  package require MergerControl
#  package require MERGERunControl


  Setting Option7Name "Project Home Page"
  Setting Option7URL  "http://npg.dl.ac.uk/NPG/Projects/Fair.html"

  Setting Option8Name "Server Software Home Page"
  Setting Option8URL  "http://npg.dl.ac.uk/MIDAS/download/WebServices.html"


  285   Wed Apr 28 10:06:11 2021 NHWednesday 28 April
11.05 All system wide checks OK *except*
WR
	
		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc971 : 	 17  
aida02 fault 	 0x8bbf : 	 0x8bd0 : 	 17  
aida03 fault 	 0xae9e : 	 0xaeaf : 	 17  
aida04 fault 	 0x6e3 : 	 0x6f4 : 	 17  
aida05 fault 	 0x31a4 : 	 0x31bd : 	 25  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x2303 : 	 25  
aida07 fault 	 0x438f : 	 0x43a8 : 	 25  
aida08 fault 	 0x61ed : 	 0x6206 : 	 25  
aida09 fault 	 0x1c32 : 	 0x1c40 : 	 14  
aida10 fault 	 0x9332 : 	 0x933f : 	 13  
aida11 fault 	 0x227b : 	 0x2288 : 	 13  
aida12 fault 	 0xb2fc : 	 0xb309 : 	 13  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA
	
			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x1 : 	 1  
aida02 fault 	 0x0 : 	 0x3 : 	 3  
aida03 fault 	 0x0 : 	 0x1 : 	 1  
aida04 fault 	 0x0 : 	 0x2 : 	 2  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida08 fault 	 0x0 : 	 0x1 : 	 1  
aida11 fault 	 0x0 : 	 0x3 : 	 3  
aida12 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 4, Failed 8
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Looks like a big WR glitch happened at some point (master in issue?)

Memory
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26     20     26     15      3      1      3      2      2      3      4   : 27848
aida02 :     25      7      1      4      1      4      2      2      3      3      6   : 36204
aida03 :     30      5      3      3      3      3      1      3      3      3      6   : 36464
aida04 :      5     10      0      5      2      3      1      3      3      3      6   : 36356
aida05 :     34     40     16     13      4      3      3      2      3      3      4   : 29160
aida06 :     16      8      4      2      1      4      1      3      3      3      6   : 36416
aida07 :     28     23     23     10      4      3      2      4      3      4      3   : 27736
aida08 :     26      9      3      3      1      3      1      3      3      3      6   : 36352
aida09 :     54     42     10     17      3      4      3      4      3      3      4   : 30376
aida10 :     17      5      7      4      3      3      3      3      2      3      6   : 35996
aida11 :     19     10     11      2      1      1      2      4      2      3      6   : 35916
aida12 :     82     42     12      7      4      2      3      2      2      3      4   : 27960


Options unchanged from ELOG 284

DAQ continues OK but New Merger window has disappeared... clearly still running in background!

DSSSD leakage currents decreasing as expected

-

Plan later today:

Prepare AIDA Kapton Cables for triple

install 4 new FEE64s into S4 (13,14,15,16)

Proposed layout of FEE64s (perspective with beam) in fig 5

Cards 9-12 will be moved on Friday when triple is installed

When ready MAC addressed will be given to OH/TD who will do the PC configuration including:
- Network IP allocation
- MIDAS registration
- Firmware updating
- Checks 

17.20:

The four new FEEs have been installed and connected to Ethernet, Power, HDMI and Water
No front-end adapter is connected yet
The MAC addresss are as follows
aida13 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d:15
aida14 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:42: d: b
aida15 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ee:10
aida16 xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ed
Attachment 1: elog_temps.png
elog_temps.png
Attachment 2: elog_gswr.png
elog_gswr.png
Attachment 3: elog_tse.png
elog_tse.png
Attachment 4: elog_hv.png
elog_hv.png
Attachment 5: AIDA_FEE.png.jpeg
AIDA_FEE.png.jpeg
  284   Tue Apr 27 08:26:01 2021 TDTuesday 27 April - aida09 system console reports 'received undefined information code: 1' and 'code: 9'
09.25 All system wide checks OK *except*


		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc963 : 	 3  
aida02 fault 	 0x8bbf : 	 0x8bc2 : 	 3  
aida03 fault 	 0xae9e : 	 0xaea1 : 	 3  
aida04 fault 	 0x6e3 : 	 0x6e6 : 	 3  
aida05 fault 	 0x31a4 : 	 0x31ac : 	 8  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x22f2 : 	 8  
aida07 fault 	 0x438f : 	 0x4397 : 	 8  
aida08 fault 	 0x61ed : 	 0x61f5 : 	 8  
aida09 fault 	 0x1c32 : 	 0x1c33 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16     18     24     16      3      1      3      2      2      3      4   : 27792
aida02 :     25      6      3      5      1      4      2      2      3      3      6   : 36260
aida03 :     19     10      3      6      3      3      1      3      3      3      6   : 36556
aida04 :     21      7      3      4      2      4      2      2      3      3      6   : 36284
aida05 :     40     37     16     13      4      3      2      2      3      3      4   : 28904
aida06 :     32      4      4      3      0      4      1      3      3      3      6   : 36416
aida07 :     26     22     18     13      3      3      2      4      3      4      3   : 27672
aida08 :     15      9      4      1      2      3      1      3      3      3      6   : 36324
aida09 :     29     41     11     19      3      4      3      4      3      3      4   : 30348
aida10 :     17      5      8      4      3      3      3      3      2      3      6   : 36012
aida11 :     25      9      3      5      1      1      2      4      2      3      6   : 35900
aida12 :     80     43     12      7      4      2      3      2      2      3      4   : 27960



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

09.25 DSSSD bias |& leakage currents Ok - attachment 1
      FEE64 temps OK - attachment 2
      good event stats - attachment 3
       aida09 zero rate
      merger time seq errors - attachment 4
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5

      DAQ continues OK *except* aida09 - no data storage

10.00 from aida09 system console

24/03:04:39|In RDOGo_Operate: Enabled Correlation, ASIC and discriminator readout
24/03:04:39|Also in RDOGo_Operate: Enabled waveform readout
24/03:04:39|
24/03:04:39|get_WAVEBlk (C) : which = 0 , Status = 00001031 , Count = 0
24/03:04:39|get_WAVEBlk (A) which = 0 ---- status = 0540, State machines = 050A
24/03:04:39|WV:0:Buffer info. Length = 131072 : Offset = 0 
24/03:04:40|WV:1:Buffer info. Length = 131070 : Offset = 2 
24/03:04:40|WV:2:Buffer info. Length = 131068 : Offset = 4 
24/03:04:40|WV:3:Buffer info. Length = 131066 : Offset = 6 
24/03:04:40|WV:4:Buffer info. Length = 131064 : Offset = 8 
24/03:04:40|WV:5:Buffer info. Length = 131062 : Offset = 10 
24/03:04:40|WV:6:Buffer info. Length = 131060 : Offset = 12 
24/03:04:40|WV:7:Buffer info. Length = 131058 : Offset = 14 
24/03:04:40|WV:8:Buffer info. Length = 131056 : Offset = 16 
24/03:04:40|WV:9:Buffer info. Length = 131054 : Offset = 18 
24/03:04:40|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:40|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => 804441A5 : 1 => 09CD0000 
24/03:04:40|Last Data : Databuffer  274139128 : 0 => C01B7F88 : 1 => 0C05B135 
24/03:04:40| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51 
24/03:04:40| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A 
24/03:04:40| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26 
24/03:04:41| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E 
24/03:04:41| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2 
24/03:04:41| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B 
24/03:04:41| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC 
24/03:04:41| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E 
24/03:04:41|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C03A7F13 : 1 => 0C05B455 
24/03:04:41|Last Data : Databuffer  274139128 : 0 => C0127D9D : 1 => 0DB9DEED 
24/03:04:41| 1FA61FB2 : 1F951F66 | 1F361F5D : 1F751F0F | 1F191EF0 : 1EF31EE9 | 1E701E8E : 1EEB1F51 
24/03:04:41| 1F521F99 : 1FF52005 | 1FE52027 : 20171FCC | 1FAD1F3D : 1ED41E7B | 1EB11E77 : 3FAB373A 
24/03:04:41| 8050023F : 0AE0C000 | 804441A5 : 0AE0C000 | 400003FC : 0AE0C4D0 | 1E581F6F : 1E451D26 
24/03:04:42| 1FB01E60 : 1D2E1F80 | 1E251D88 : 1FA81F4A | 1DD51F1F : 1E1D1DF7 | 1EBC1DF0 : 1DF41E7E 
24/03:04:42| 1DEA1DF5 : 1E9E1E83 | 1E231E9C : 1E661E60 | 1EA41E6F : 1E101D9C | 1D751D92 : 1DFC1DF2 
24/03:04:42| 1E201E47 : 1E7C1E6C | 1E351E00 : 1DD91DFE | 1DD81DB5 : 1DE91E1C | 1E261DD6 : 1E1A1E0B 
24/03:04:42| 1DFB1DCC : 1DEA1E11 | 1E201E39 : 1E351E7A | 1E7B1DBE : 1E7A1E81 | 1E261DFC : 1DEA1DAC 
24/03:04:42| 1D831DAC : 1DF51E1F | 1E4E1E48 : 1E421E50 | 1E861DDB : 1E361E18 | 1D551DB9 : 1DC51D8E 
24/03:04:42|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:42|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C02D7E26 : 1 => 0DB9DEED 
24/03:04:42|Last Data : Databuffer  274139128 : 0 => C0217F73 : 1 => 0F816B3D 
24/03:04:42| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:42| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:42| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:43| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:43| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:43| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:43| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:43| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:43|get_ASICBlk : Blk=0 : bytes = 1048576 : Blk_Status = 00000048 : Offset = 00000000 : Databuffer : 273090556 : 0 => C02D7EB4 : 1 => 0F816D95 
24/03:04:43|Last Data : Databuffer  274139128 : 0 => C0167F53 : 1 => 0148F2DD 
24/03:04:43| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:43| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:43| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:44| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:44| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:44| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:44| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:44|get_ASICBlk : Blk=1 : bytes = 1048576 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0207E17 : 1 => 0148F2DD 
24/03:04:44|Last Data : Databuffer  274139128 : 0 => C0177F46 : 1 => 047FF0CD 
24/03:04:44| 1EFB1F2D : 1F311F1E | 1F3B1F1C : 1EDD1EC4 | 1ED71EE6 : 1F301F09 | 1F4D1FAA : 1F1E1F81 
24/03:04:44| 1F491F0C : 1F481F31 | 1F6C1F7E : 1F601F8E | 1FC71FA9 : 1F451F9F | 1F8C1F1D : 1F551FA9 
24/03:04:44| 1F671E98 : 1F421F47 | 1ED91F67 : 1F911F70 | 1DFD1F98 : 224A1D5B | 1CF821BC : 1F771E94 
24/03:04:44| 213F1EE7 : 1DBC20F2 | 1F961D9F : 1FC81FE3 | 1EA0206E : 1F9D1DD9 | 20042040 : 1E6A1ECA 
24/03:04:45| 1F7E1FD0 : 1F981EC7 | 1E7A1F1B : 1F5F1F27 | 1F121F10 : 1F531F92 | 1F5B1EF8 : 1FA12003 
24/03:04:45| 1F7A1F97 : 1F561F26 | 200C1F80 : 1FC3201A | 1FA81FA4 : 1FEB1F99 | 1F5C1F9D : 1F8F1F98 
24/03:04:45| 1FCC1FC5 : 1FB01F92 | 1FA21F6D : 1F261F0A | 1F141F1C : 1F461F24 | 1EEE1FDB : 20221F2F 
24/03:04:45| 1F741FF1 : 1F891F8A | 1FC71F5E : 1F961FF2 | 1F6B1F6C : 1FB11F74 | 1F821FA8 : 1F841F54 
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:45|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:04:47|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:04:56|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:05|get_WAVEBlk (A) which = 0 ---- status = 0048, State machines = 0908
24/03:05:15|get_WAVEBlk (A) which = 1 ---- status = 0088, State machines = 0908
24/03:05:24|197 spectra zeroed
25/10:21:10|do_GetState returned z=0 and 1
25/10:21:11|Data Acquisition Statistics counters now cleared
25/10:24:01|Clear Statistics (1)
25/10:24:01|bad event data (0x45000034 0x84e64000)
26/16:52:50|bad event data (0x40061e21 0xc0a80b63)
26/16:52:50|received undefined information code: 1
26/16:52:50|bad event data (0x0101080a 0xe9b68def)
26/16:52:50|bad event data (0x03527827 0x0101050a)
26/16:52:50|received undefined information code: 9

From system console

Apr 24 02:52:42 aida09 kernel: IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
Apr 24 02:52:42 aida09 kernel: IP-Config: Complete:
Apr 24 02:52:42 aida09 kernel:      device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
Apr 24 02:52:42 aida09 kernel:      host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
Apr 24 02:52:42 aida09 kernel:      bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100003/2 on 192.168.11.99
Apr 24 02:52:42 aida09 kernel: Looking up port of RPC 100005/1 on 192.168.11.99
Apr 24 02:52:43 aida09 kernel: VFS: Mounted root (nfs filesystem) on device 0:12.
Apr 24 02:52:43 aida09 kernel: Freeing unused kernel memory: 168k init
Apr 24 02:52:43 aida09 kernel: xaida: device parameters: base=0x81000000 size=0x200000
Apr 24 02:52:44 aida09 kernel: Trying to free nonexistent resource <0000000081000000-00000000811fffff>
Apr 24 02:52:44 aida09 kernel: xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
Apr 24 02:52:44 aida09 kernel: xaida: driver assigned major number 254
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service telnet, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: IPv6 socket creation failed for service ftp, trying IPv4
Apr 24 02:52:44 aida09 xinetd[273]: xinetd Version 2.3.14 started with libwrap loadavg options compiled in.
Apr 24 02:52:44 aida09 xinetd[273]: Started working: 2 available services
Apr 24 02:52:48 aida09 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Apr 24 02:52:48 aida09 kernel: AIDAMEM: aidamem: driver assigned major number 253
Apr 24 02:53:04 aida09 kernel: xaida: open:
Apr 24 02:53:04 aida09 kernel: AIDAMEM: aidamem_open:
Apr 24 02:57:33 aida09 kernel: xaida: open:
Apr 24 13:39:36 aida09 xinetd[273]: START: telnet pid=396 from=192.168.11.99
Apr 24 13:42:05 aida09 xinetd[273]: EXIT: telnet status=0 pid=396 duration=149(sec)
Apr 27 09:01:50 aida09 xinetd[273]: START: telnet pid=422 from=192.168.11.99
Attachment 1: 303.png
303.png
Attachment 2: Screenshot_2021-04-27_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-27_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-27_Statistics_aidas-gsi.png
Screenshot_2021-04-27_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-27_GSI_White_Rabbit_Timestamp_Control_aidas-gsi.png
Screenshot_2021-04-27_GSI_White_Rabbit_Timestamp_Control_aidas-gsi.png
Attachment 5: Screenshot_2021-04-27_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-27_NewMerger_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-04-27_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-27_AIDA_Alerting_-_Grafana.png
  283   Mon Apr 26 09:41:21 2021 TDMonday 26 April
10.42 All system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0xc960 : 	 0xc962 : 	 2  
aida02 fault 	 0x8bbf : 	 0x8bc1 : 	 2  
aida03 fault 	 0xae9e : 	 0xaea0 : 	 2  
aida04 fault 	 0x6e3 : 	 0x6e5 : 	 2  
aida05 fault 	 0x31a4 : 	 0x31ab : 	 7  
aida05 : WR status 0x10
 aida06 fault 	 0x22ea : 	 0x22f1 : 	 7  
aida07 fault 	 0x438f : 	 0x4396 : 	 7  
aida08 fault 	 0x61ed : 	 0x61f4 : 	 7  
aida09 fault 	 0x1c32 : 	 0x1c33 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     25      8     28      8      3      2      2      3      2      3      4   : 27940
aida02 :     24      9      3      5      1      3      1      3      3      3      6   : 36408
aida03 :     24      6      5      4      3      3      1      3      3      3      6   : 36512
aida04 :     23      6      6      1      3      3      1      3      3      3      6   : 36428
aida05 :     15     21     27      7      4      2      2      3      3      3      4   : 29044
aida06 :     24      8      2      4      1      3      1      3      3      3      6   : 36352
aida07 :     29     24     28      9      3      3      2      2      2      3      4   : 27732
aida08 :     17     10      1      1      1      4      1      3      3      3      6   : 36356
aida09 :     41     49     18     14      3      4      3      4      2      3      4   : 29388
aida10 :      3      1      6      4      2      3      3      3      2      3      6   : 35828
aida11 :     22      7      5      4      2      1      2      4      2      3      6   : 35936
aida12 :     65     30     23      9      3      1      2      3      2      3      4   : 28100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021


10.47 DSSSD bias |& leakage currents Ok - attachment 1
      FEE64 temps OK - attachment 2
      good event stats - attachment 3
      merger time seq errors - attachment 4
      Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 5

      DAQ continues OK - no data storage
Attachment 1: 302.png
302.png
Attachment 2: Screenshot_2021-04-26_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-26_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-26_Statistics_aidas-gsi.png
Screenshot_2021-04-26_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-26_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-26_NewMerger_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-26_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-26_AIDA_Alerting_-_Grafana.png
  282   Sun Apr 25 10:22:16 2021 TDSunday 25 April
11.25 All system wide checks OK *except*

	 Base 		Current 	Difference
aida01 fault 	 0xc95f : 	 0xc960 : 	 1  
aida02 fault 	 0x8bbe : 	 0x8bbf : 	 1  
aida03 fault 	 0xae9d : 	 0xae9e : 	 1  
aida04 fault 	 0x6e2 : 	 0x6e3 : 	 1  
aida05 fault 	 0x31a2 : 	 0x31a4 : 	 2  
aida05 : WR status 0x10
 aida06 fault 	 0x22e7 : 	 0x22ea : 	 3  
aida07 fault 	 0x438d : 	 0x438f : 	 2  
aida08 fault 	 0x61eb : 	 0x61ed : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     55     21     31      6      3      2      2      3      2      3      4   : 28148
aida02 :     28      3      5      4      0      3      1      3      3      3      6   : 36312
aida03 :     25     10      2      2      2      4      1      3      3      3      6   : 36500
aida04 :     14      7     10      1      2      2      2      3      3      3      6   : 36528
aida05 :     58     35     28      6      4      2      2      3      3      3      4   : 29312
aida06 :     16     11      6      1      0      4      1      3      3      3      6   : 36376
aida07 :     75     31     25      9      4      3      2      2      2      3      4   : 27988
aida08 :     28      7      1      1      1      4      1      3      3      3      6   : 36376
aida09 :     50     48     28     10      3      3      3      4      2      3      4   : 29320
aida10 :     26      5      4      3      1      3      2      4      2      3      6   : 36080
aida11 :     21      7      5      3      5      2      2      4      2      3      6   : 36220
aida12 :     64     29     26      7      3      1      2      3      2      3      4   : 28072

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Apr 24 03:57:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

Most recent merger errors reported - server terminal now reset & clear'd

MERGE Data Link (4408): bad timestamp  10 3 0xc2b67d8f 0x0b50c3e2 0x0000f437cb50c3e2 0x1678f437cb50c3e2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2877f27 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2a07fcd 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2b77c9c 0x0b50cbb2 0x0000f437cb50cbb2 0x1678f437cb50cbb2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc28b7d9c 0x0b50d382 0x0000f437cb50d382 0x1678f437cb50d382 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2937e6c 0x0b50e322 0x0000f437cb50e322 0x1678f437cb50e322 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2a17d78 0x0b50e322 0x0000f437cb50e322 0x1678f437cb50e322 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2817fff 0x0b50eaf2 0x0000f437cb50eaf2 0x1678f437cb50eaf2 0x1678f43a16f1e6b2
MERGE Data Link (4408): bad timestamp  10 3 0xc2b77ecf 0x06fb3d52 0x0000f437cb50eaf2 0x1678f437c6fb3d52 0x1678f43a16f1e6b2

11.25 Grafana DSSSD bias |& leakage currents for previous 7 days - attachment 1
      DSSSD bias |& leakage currents Ok - attachment 2
      FEE64 temps OK - attachment 3
      good event stats - attachment 4
      merger time seq errors - attachment 5

      All histograms, stats and merge stats zero'd
      WR decoder errors *new*  baseline

11.45 1.8.L spectra - attachments 6 & 7
      peak width FWHM aida05 65 ch, aida06 95 ch
      1.8.W spectra = attachments 8 & 9
Attachment 1: Screenshot_2021-04-25_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-25_AIDA_Alerting_-_Grafana.png
Attachment 2: 300.png
300.png
Attachment 3: Screenshot_2021-04-25_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-25_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2021-04-25_Statistics_aidas-gsi.png
Screenshot_2021-04-25_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-25_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-25_NewMerger_Statistics_aidas-gsi.png
Attachment 6: Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi.png
Attachment 7: Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(2).png
Attachment 9: Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-25_Spectrum_Browser_aidas-gsi(3).png
  281   Sat Apr 24 15:48:38 2021 TDaida12 crash - 08:33:03 Monday 19 April
from ttyUSB5 system console log

19/07:25:59|page:c047b060 flags:(null) count:-1 mapcount:0 mapping:(null) index:0
19/07:25:59|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:00|Faulting instruction address: 0xc006e814
19/07:26:00|Oops: Kernel access of bad area, sig: 11 [#1]
19/07:26:00|PREEMPT Xilinx Virtex440
19/07:26:00|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:00|NIP: c006e814 LR: c006e78c CTR: c026327c
19/07:26:00|REGS: c60f3860 TRAP: 0300   Not tainted  (2.6.31)
19/07:26:00|MSR: 00021000 <ME,CE>  CR: 28222024  XER: 20000000
19/07:26:00|DEAR: 00100104, ESR: 00800000
19/07:26:00|TASK = c6b0a0a0[224] 'syslogd' THREAD: c60f2000
19/07:26:00|GPR00: 00000000 c60f3910 c6b0a0a0 c038c3a8 00000000 000001a5 c047b078 00000041
19/07:26:00|GPR08: 00000001 00100100 00000004 c038c3a8 48282028 2015881c 00000001 00000000
19/07:26:00|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c60f2000
19/07:26:00|GPR24: 00000000 00000001 00029000 c038c384 00000001 c047b060 c038c39c c038c394
19/07:26:00|NIP [c006e814] get_page_from_freelist+0x188/0x4cc
19/07:26:00|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:00|Call Trace:
19/07:26:00|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:00|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:01|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:01|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:01|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:01|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:01|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:01|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:01|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:01|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:01|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:01|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:01|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:01|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:01|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:01|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:01|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:01|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:01|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:01|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:01|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:01|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:02|Instruction dump:
19/07:26:02|2f890000 419e0008 7c004a2c 38dd0018 7f861800 409effd8 7f861800 419e02e0
19/07:26:02|813d0018 81660004 7f3ccb78 912b0000 <91690004> 92dd0018 92a60004 813e0000
19/07:26:02|---[ end trace 87f7fdd1c04ff304 ]---
19/07:26:02|note: syslogd[224] exited with preempt_count 1
19/07:26:02|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:02|Faulting instruction address: 0xc006e388
19/07:26:02|Oops: Kernel access of bad area, sig: 11 [#2]
19/07:26:02|PREEMPT Xilinx Virtex440
19/07:26:02|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:02|NIP: c006e388 LR: c006e348 CTR: c01c13cc
19/07:26:02|REGS: c60f35d0 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:02|MSR: 00021000 <ME,CE>  CR: 28222082  XER: 00000000
19/07:26:02|DEAR: 00100104, ESR: 00800000
19/07:26:02|TASK = c6b0a0a0[224] 'syslogd' THREAD: c60f2000
19/07:26:02|GPR00: 00000000 c60f3680 c6b0a0a0 00000002 00000003 c038c3a8 ffffffff c0498000
19/07:26:02|GPR08: 00000008 c03c7738 00000006 00100100 00000000 2015881c c69e63f8 1fd84000
19/07:26:02|GPR16: c0390000 c60f2000 c69e63f8 c6ba2128 c6178740 c036f924 ffffffff 00000000
19/07:26:03|GPR24: 1fd84000 00000000 c038c384 c038c384 00100008 00021000 c038c39c c03c7720
19/07:26:03|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:03|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:03|Call Trace:
19/07:26:03|[c60f3680] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:03|[c60f36b0] [c0072db0] put_page+0x50/0x1a0
19/07:26:03|[c60f36d0] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:03|[c60f36e0] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:03|[c60f3770] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:03|[c60f37a0] [c0032348] mmput+0x50/0x10c
19/07:26:03|[c60f37b0] [c0036554] exit_mm+0x10c/0x134
19/07:26:03|[c60f37e0] [c0037e38] do_exit+0xd4/0x61c
19/07:26:03|[c60f3820] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:03|[c60f3840] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:03|[c60f3850] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:03|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:03|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:03|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:03|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:04|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:04|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:04|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:04|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:04|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:04|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:04|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:04|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:04|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:04|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:04|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:04|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:04|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:04|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:04|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:04|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:04|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:04|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:04|Instruction dump:
19/07:26:04|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:05|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:05|---[ end trace 87f7fdd1c04ff305 ]---
19/07:26:05|Fixing recursive fault but reboot is needed!
19/07:26:05|BUG: scheduling while atomic: syslogd/224/0x00000006
19/07:26:05|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:05|Call Trace:
19/07:26:05|[c60f34c0] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:05|[c60f3500] [c002d41c] __schedule_bug+0x68/0x7c
19/07:26:05|[c60f3510] [c02a8020] schedule+0x2f4/0x34c
19/07:26:05|[c60f3550] [c00382a8] do_exit+0x544/0x61c
19/07:26:05|[c60f3590] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:05|[c60f35b0] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:05|[c60f35c0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:05|[c60f3680] [c006e324] free_hot_cold_page+0xa8/0x234
19/07:26:05|[c60f36b0] [c0072db0] put_page+0x50/0x1a0
19/07:26:05|[c60f36d0] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:05|[c60f36e0] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:05|[c60f3770] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:05|[c60f37a0] [c0032348] mmput+0x50/0x10c
19/07:26:05|[c60f37b0] [c0036554] exit_mm+0x10c/0x134
19/07:26:06|[c60f37e0] [c0037e38] do_exit+0xd4/0x61c
19/07:26:06|[c60f3820] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:06|[c60f3840] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:06|[c60f3850] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:06|[c60f3910] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:06|[c60f39a0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:06|[c60f3a20] [c0242318] ip_append_data+0x558/0x974
19/07:26:06|[c60f3aa0] [c0263584] udp_sendmsg+0x308/0x61c
19/07:26:06|[c60f3b50] [c026b5c8] inet_sendmsg+0x4c/0x78
19/07:26:06|[c60f3b70] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:06|[c60f3c60] [c020f9b0] kernel_sendmsg+0x2c/0x44
19/07:26:06|[c60f3c80] [c02916ac] xs_send_kvec+0x74/0xa8
19/07:26:06|[c60f3cc0] [c029189c] xs_sendpages+0x1bc/0x250
19/07:26:06|[c60f3cf0] [c0293518] xs_udp_send_request+0x64/0x180
19/07:26:06|[c60f3d10] [c0290284] xprt_transmit+0x70/0x254
19/07:26:06|[c60f3d40] [c028df84] call_transmit+0x190/0x2ac
19/07:26:06|[c60f3d60] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:06|[c60f3da0] [c028d120] rpc_run_task+0x44/0x90
19/07:26:06|[c60f3db0] [c013a584] nfs_write_rpcsetup+0x180/0x20c
19/07:26:06|[c60f3e20] [c013632c] nfs_pageio_doio+0x70/0xa0
19/07:26:07|[c60f3e30] [c013ade0] nfs_writepages+0xd8/0x128
19/07:26:07|[c60f3e90] [c006fe38] do_writepages+0x4c/0x84
19/07:26:07|[c60f3ea0] [c0068f78] __filemap_fdatawrite_range+0x7c/0x90
19/07:26:07|[c60f3f00] [c00bca24] vfs_fsync+0x60/0xe4
19/07:26:07|[c60f3f20] [c00bcad8] do_fsync+0x30/0x54
19/07:26:07|[c60f3f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:07|Unable to handle kernel paging request for data at address 0x00100100
19/07:26:07|Faulting instruction address: 0xc006e9e8
19/07:26:07|Oops: Kernel access of bad area, sig: 11 [#3]
19/07:26:07|PREEMPT Xilinx Virtex440
19/07:26:07|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:07|NIP: c006e9e8 LR: c006e78c CTR: c0285b38
19/07:26:07|REGS: c61adab0 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:07|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:26:07|DEAR: 00100100, ESR: 00000000
19/07:26:07|TASK = c6b0a4c0[227] 'klogd' THREAD: c61ac000
19/07:26:07|GPR00: 00000010 c61adb60 c6b0a4c0 c038c3a8 00000000 000001a5 00000000 00000041
19/07:26:07|GPR08: 00000001 00100100 00000004 c61ac000 22024048 2079d364 00000001 00000000
19/07:26:08|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c61ac000
19/07:26:08|GPR24: 00000000 00000001 00029000 c038c384 c038cc18 001000e8 c038c39c c038c394
19/07:26:08|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:08|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:08|Call Trace:
19/07:26:08|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:08|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:08|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:08|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:08|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:08|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:08|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:08|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:08|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:08|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:08|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:08|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:08|Instruction dump:
19/07:26:08|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:09|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:09|---[ end trace 87f7fdd1c04ff306 ]---
19/07:26:09|note: klogd[227] exited with preempt_count 1
19/07:26:09|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:09|Faulting instruction address: 0xc006e388
19/07:26:09|Oops: Kernel access of bad area, sig: 11 [#4]
19/07:26:09|PREEMPT Xilinx Virtex440
19/07:26:09|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:09|NIP: c006e388 LR: c006e348 CTR: c01c13cc
19/07:26:09|REGS: c61ad820 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:09|MSR: 00021000 <ME,CE>  CR: 22224082  XER: 00000000
19/07:26:09|DEAR: 00100104, ESR: 00800000
19/07:26:09|TASK = c6b0a4c0[227] 'klogd' THREAD: c61ac000
19/07:26:09|GPR00: 00000000 c61ad8d0 c6b0a4c0 00000002 00000003 c038c3a8 ffffffff c0498000
19/07:26:09|GPR08: 00000008 c03c7e98 00000006 00100100 00000000 2079d364 c6a5640c 20739000
19/07:26:09|GPR16: c0390000 c61ac000 c6a5640c c60a6320 c6178c80 c036f924 ffffffff 00000000
19/07:26:09|GPR24: 20739000 00000000 c038c384 c038c384 00100008 00021000 c038c39c c03c7e80
19/07:26:10|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:10|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:10|Call Trace:
19/07:26:10|[c61ad8d0] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:10|[c61ad900] [c0072db0] put_page+0x50/0x1a0
19/07:26:10|[c61ad920] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:10|[c61ad930] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:10|[c61ad9c0] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:10|[c61ad9f0] [c0032348] mmput+0x50/0x10c
19/07:26:10|[c61ada00] [c0036554] exit_mm+0x10c/0x134
19/07:26:10|[c61ada30] [c0037e38] do_exit+0xd4/0x61c
19/07:26:10|[c61ada70] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:10|[c61ada90] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:10|[c61adaa0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:10|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:10|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:10|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:10|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:10|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:10|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:11|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:11|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:11|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:11|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:11|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:11|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:11|Instruction dump:
19/07:26:11|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:11|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:11|---[ end trace 87f7fdd1c04ff307 ]---
19/07:26:11|Fixing recursive fault but reboot is needed!
19/07:26:11|BUG: scheduling while atomic: klogd/227/0x00000006
19/07:26:11|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:11|Call Trace:
19/07:26:11|[c61ad710] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:11|[c61ad750] [c002d41c] __schedule_bug+0x68/0x7c
19/07:26:11|[c61ad760] [c02a8020] schedule+0x2f4/0x34c
19/07:26:11|[c61ad7a0] [c00382a8] do_exit+0x544/0x61c
19/07:26:11|[c61ad7e0] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:11|[c61ad800] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:12|[c61ad810] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:12|[c61ad8d0] [c006e324] free_hot_cold_page+0xa8/0x234
19/07:26:12|[c61ad900] [c0072db0] put_page+0x50/0x1a0
19/07:26:12|[c61ad920] [c008c918] free_page_and_swap_cache+0x34/0x8c
19/07:26:12|[c61ad930] [c007f57c] unmap_vmas+0x2bc/0x6ac
19/07:26:12|[c61ad9c0] [c0084330] exit_mmap+0xc8/0x1a4
19/07:26:12|[c61ad9f0] [c0032348] mmput+0x50/0x10c
19/07:26:12|[c61ada00] [c0036554] exit_mm+0x10c/0x134
19/07:26:12|[c61ada30] [c0037e38] do_exit+0xd4/0x61c
19/07:26:12|[c61ada70] [c000bd58] kernel_bad_stack+0x0/0x4c
19/07:26:12|[c61ada90] [c001191c] bad_page_fault+0x90/0xd8
19/07:26:12|[c61adaa0] [c000e834] handle_page_fault+0x7c/0x80
19/07:26:12|[c61adb60] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:12|[c61adbf0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:12|[c61adc70] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:12|[c61adcd0] [c009270c] __kmalloc+0x118/0x120
19/07:26:12|[c61adcf0] [c0216fec] __alloc_skb+0x58/0x138
19/07:26:12|[c61add10] [c0211818] sock_alloc_send_pskb+0x1ec/0x338
19/07:26:12|[c61add60] [c0285d38] unix_dgram_sendmsg+0x200/0x564
19/07:26:13|[c61addd0] [c020e2c0] sock_aio_write+0xf0/0x104
19/07:26:13|[c61ade30] [c0097254] do_sync_write+0xc4/0x138
19/07:26:13|[c61adef0] [c0097dc4] vfs_write+0x180/0x188
19/07:26:13|[c61adf10] [c0098388] sys_write+0x4c/0x90
19/07:26:13|[c61adf40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:13|Unable to handle kernel paging request for data at address 0x00000000
19/07:26:13|Faulting instruction address: 0xc006e9e8
19/07:26:13|Oops: Kernel access of bad area, sig: 11 [#5]
19/07:26:13|PREEMPT Xilinx Virtex440
19/07:26:13|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:13|NIP: c006e9e8 LR: c006e78c CTR: c006b2c4
19/07:26:13|REGS: c6349900 TRAP: 0300   Tainted: G      D     (2.6.31)
19/07:26:13|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:26:13|DEAR: 00000000, ESR: 00000000
19/07:26:13|TASK = c62fa500[379] 'AidaExecV9' THREAD: c6348000
19/07:26:13|GPR00: 00000009 c63499b0 c62fa500 c038c3a8 00000000 000001a5 00100100 00000041
19/07:26:13|GPR08: 00000001 00000000 00000004 c6348000 22022048 1005520c 00000001 00000000
19/07:26:13|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c6348000
19/07:26:14|GPR24: 00000000 00000001 00021000 c038c384 c038cc18 ffffffe8 c038c39c c038c394
19/07:26:14|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:14|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:14|Call Trace:
19/07:26:14|[c63499b0] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:14|[c6349a40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:14|[c6349ac0] [c0092360] cache_alloc_refill+0x374/0x608
19/07:26:14|[c6349b20] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:26:14|[c6349b40] [c006b2e0] mempool_alloc_slab+0x1c/0x2c
19/07:26:14|[c6349b50] [c006b5d8] mempool_alloc+0x5c/0x13c
19/07:26:14|[c6349ba0] [c0294600] rpc_malloc+0x34/0x9c
19/07:26:14|[c6349bc0] [c028da44] call_allocate+0xc4/0x194
19/07:26:14|[c6349be0] [c0294e00] __rpc_execute+0xbc/0x300
19/07:26:14|[c6349c20] [c028d120] rpc_run_task+0x44/0x90
19/07:26:14|[c6349c30] [c028d234] rpc_call_sync+0x54/0x84
19/07:26:14|[c6349c60] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
19/07:26:14|[c6349c90] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
19/07:26:14|[c6349d50] [c0129b3c] nfs_permission+0x178/0x1c4
19/07:26:14|[c6349d70] [c00a0bf4] inode_permission+0x94/0xbc
19/07:26:15|[c6349d80] [c00a2bc0] __link_path_walk+0xb0/0xf7c
19/07:26:15|[c6349df0] [c00a3ad4] path_walk+0x48/0xc8
19/07:26:15|[c6349e20] [c00a3c44] do_path_lookup+0x64/0x6c
19/07:26:15|[c6349e40] [c00a4b14] do_filp_open+0xc8/0x914
19/07:26:15|[c6349f10] [c0095314] do_sys_open+0x64/0x130
19/07:26:15|[c6349f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:15|Instruction dump:
19/07:26:15|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:15|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:15|---[ end trace 87f7fdd1c04ff308 ]---
19/07:26:15|note: AidaExecV9[379] exited with preempt_count 1
19/07:26:15|Call Trace:
19/07:26:15|[c6311b50] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:26:15|[c6311b90] [c006da9c] bad_page+0x94/0x12c
19/07:26:15|[c6311bb0] [c006eadc] get_page_from_freelist+0x450/0x4cc
19/07:26:15|[c6311c40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:15|[c6311cc0] [c024cd9c] tcp_sendmsg+0xb34/0xbe4
19/07:26:15|[c6311d40] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:15|[c6311e30] [c020f944] sys_sendto+0xbc/0xf0
19/07:26:15|[c6311f00] [c02103b8] sys_socketcall+0x160/0x1f8
19/07:26:16|[c6311f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:16|Unable to handle kernel paging request for data at address 0x00000000
19/07:26:16|Faulting instruction address: 0xc006e9e8
19/07:26:16|Oops: Kernel access of bad area, sig: 11 [#6]
19/07:26:16|PREEMPT Xilinx Virtex440
19/07:26:16|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:16|NIP: c006e9e8 LR: c006e78c CTR: c01c13cc
19/07:26:16|REGS: c6311b00 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:26:16|MSR: 00021000 <ME,CE>  CR: 24222024  XER: 00000000
19/07:26:16|DEAR: 00000000, ESR: 00000000
19/07:26:16|TASK = c62fb9a0[375] 'AidaExecV9' THREAD: c6310000
19/07:26:16|GPR00: 00000009 c6311bb0 c62fb9a0 c038c3a8 c62fb9a0 00000010 00100100 0000594b
19/07:26:16|GPR08: 000000a0 00000000 00000000 c6310000 24222042 1005520c 00000001 00000000
19/07:26:16|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c6310000
19/07:26:16|GPR24: 00000000 00000001 00029000 c038c384 00000001 ffffffe8 c038c39c c038c394
19/07:26:16|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:26:16|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:26:17|Call Trace:
19/07:26:17|[c6311bb0] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:26:17|[c6311c40] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:26:17|[c6311cc0] [c024cd9c] tcp_sendmsg+0xb34/0xbe4
19/07:26:17|[c6311d40] [c020f5cc] sock_sendmsg+0xac/0xe4
19/07:26:17|[c6311e30] [c020f944] sys_sendto+0xbc/0xf0
19/07:26:17|[c6311f00] [c02103b8] sys_socketcall+0x160/0x1f8
19/07:26:17|[c6311f40] [c000e364] ret_from_syscall+0x0/0x3c
19/07:26:17|Instruction dump:
19/07:26:17|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:26:17|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:26:17|---[ end trace 87f7fdd1c04ff309 ]---
19/07:26:17|note: AidaExecV9[375] exited with preempt_count 1
19/07:26:17|Unable to handle kernel paging request for data at address 0x00100104
19/07:26:18|Faulting instruction address: 0xc006e388
19/07:26:18|Oops: Kernel access of bad area, sig: 11 [#7]
19/07:26:18|PREEMPT Xilinx Virtex440
19/07:26:18|Modules linked in: aidamem xdriver xh_spidev_register
19/07:26:18|NIP: c006e388 LR: c006e348 CTR: 00000000
19/07:26:18|REGS: c6835e10 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:26:18|MSR: 00021000 <ME,CE>  CR: 24000082  XER: 00000000
19/07:26:18|DEAR: 00100104, ESR: 00800000
19/07:26:18|TASK = c6824880[5] 'events/0' THREAD: c6834000
19/07:26:18|GPR00: 00000000 c6835ec0 c6824880 00000000 00000003 c038c3a8 ffffffff c0498000
19/07:26:18|GPR08: 00000008 c0419a78 00000027 00100100 002515ba 00005aa8 bfffffff 5fffffff
19/07:26:18|GPR16: ffff7fd7 fffeffaf 00120110 7ffefd7f c031aef4 c02fb040 c02fb008 00000002
19/07:26:18|GPR24: c6834000 00000000 c038c384 c038c384 00000000 00029000 c038c39c c0419a60
19/07:26:18|NIP [c006e388] free_hot_cold_page+0x10c/0x234
19/07:26:18|LR [c006e348] free_hot_cold_page+0xcc/0x234
19/07:26:18|Call Trace:
19/07:26:18|[c6835ec0] [c006e324] free_hot_cold_page+0xa8/0x234 (unreliable)
19/07:26:18|[c6835ef0] [c00915c4] kmem_freepages+0xd8/0x134
19/07:26:19|[c6835f00] [c0091ab8] slab_destroy+0x34/0x8c
19/07:26:19|[c6835f10] [c0091e20] drain_freelist+0x9c/0x110
19/07:26:19|[c6835f40] [c0093158] cache_reap+0x104/0x138
19/07:26:19|[c6835f60] [c0048908] worker_thread+0x140/0x204
19/07:26:19|[c6835fc0] [c004cfc8] kthread+0x78/0x7c
19/07:26:19|[c6835ff0] [c000e140] kernel_thread+0x4c/0x68
19/07:26:19|Instruction dump:
19/07:26:19|408200fc 3d20c038 3969c3d4 812b001c 2f990000 39290001 912b001c 409e0068
19/07:26:19|817e000c 393f0018 38be000c 917f0018 <912b0004> 90a90004 913e000c 813e0000
19/07:26:19|---[ end trace 87f7fdd1c04ff30a ]---
19/07:26:19|note: events/0[5] exited with preempt_count 1
19/07:26:19|Unable to handle kernel paging request for data at address 0x00100100
19/07:29:19|Faulting instruction address: 0xc006e9e8
19/07:29:19|Oops: Kernel access of bad area, sig: 11 [#8]
19/07:29:19|PREEMPT Xilinx Virtex440
19/07:29:19|Modules linked in: aidamem xdriver xh_spidev_register
19/07:29:19|NIP: c006e9e8 LR: c006e78c CTR: c025c424
19/07:29:19|REGS: c0391a60 TRAP: 0300   Tainted: G    B D     (2.6.31)
19/07:29:19|MSR: 00021000 <ME,CE>  CR: 22222044  XER: 20000000
19/07:29:19|DEAR: 00100100, ESR: 00000000
19/07:29:19|TASK = c036e318[0] 'swapper' THREAD: c0390000
19/07:29:19|GPR00: 00000010 c0391b10 c036e318 c038c3a8 00000000 000001a5 00000000 00000041
19/07:29:19|GPR08: 00000001 00100100 00000004 c0390000 22022048 00005aa8 00000001 00000000
19/07:29:19|GPR16: 00000041 00000000 c038cc1c c0390000 c038c5ec 00200200 00100100 c0390000
19/07:29:19|GPR24: 00000000 00000001 00021000 c038c384 c038cc18 001000e8 c038c39c c038c394
19/07:29:19|NIP [c006e9e8] get_page_from_freelist+0x35c/0x4cc
19/07:29:19|LR [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:29:19|Call Trace:
19/07:29:19|[c0391b10] [c006e78c] get_page_from_freelist+0x100/0x4cc (unreliable)
19/07:29:20|[c0391ba0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:29:20|[c0391c20] [c0092360] cache_alloc_refill+0x374/0x608
19/07:29:20|[c0391c80] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:29:20|[c0391ca0] [c025c4c8] tcp_v4_conn_request+0xa4/0x408
19/07:29:20|[c0391cf0] [c02557bc] tcp_rcv_state_process+0x18c/0x9a4
19/07:29:20|[c0391d10] [c025c2b8] tcp_v4_do_rcv+0x9c/0x1b4
19/07:29:20|[c0391d40] [c025dfb8] tcp_v4_rcv+0x5ac/0x704
19/07:29:20|[c0391d70] [c023edc8] ip_local_deliver_finish+0x84/0x1f0
19/07:29:20|[c0391d90] [c023eb10] ip_rcv_finish+0x164/0x398
19/07:29:20|[c0391dc0] [c021f6c8] netif_receive_skb+0x26c/0x378
19/07:29:20|[c0391df0] [c0222518] process_backlog+0x88/0xd8
19/07:29:20|[c0391e20] [c02228a8] net_rx_action+0x8c/0x17c
19/07:29:20|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
19/07:29:20|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
19/07:29:20|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
19/07:29:20|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
19/07:29:20|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
19/07:29:20|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
19/07:29:20|[c0391fb0] [c000172c] rest_init+0x70/0x84
19/07:29:21|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
19/07:29:21|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
19/07:29:21|Instruction dump:
19/07:29:21|70080004 4182ff74 4823979d 4bffff6c 813e000c 387f0014 3ba9ffe8 48000014
19/07:29:21|801d000c 7f80c000 41befe1c 3ba9ffe8 <813d0018> 2f890000 419e0008 7c004a2c
19/07:29:21|Kernel panic - not syncing: Fatal exception in interrupt
19/07:29:21|Call Trace:
19/07:29:21|[c0391990] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/07:29:21|[c03919d0] [c00345bc] panic+0x94/0x168
19/07:29:21|[c0391a20] [c000bd44] die+0x178/0x18c
19/07:29:21|[c0391a40] [c001191c] bad_page_fault+0x90/0xd8
19/07:29:21|[c0391a50] [c000e834] handle_page_fault+0x7c/0x80
19/07:29:21|[c0391b10] [c006e78c] get_page_from_freelist+0x100/0x4cc
19/07:29:21|[c0391ba0] [c006ec30] __alloc_pages_nodemask+0xd8/0x4f8
19/07:29:21|[c0391c20] [c0092360] cache_alloc_refill+0x374/0x608
19/07:29:21|[c0391c80] [c00927d8] kmem_cache_alloc+0xc4/0xcc
19/07:29:21|[c0391ca0] [c025c4c8] tcp_v4_conn_request+0xa4/0x408
19/07:29:21|[c0391cf0] [c02557bc] tcp_rcv_state_process+0x18c/0x9a4
19/07:29:21|[c0391d10] [c025c2b8] tcp_v4_do_rcv+0x9c/0x1b4
19/07:29:22|[c0391d40] [c025dfb8] tcp_v4_rcv+0x5ac/0x704
19/07:29:22|[c0391d70] [c023edc8] ip_local_deliver_finish+0x84/0x1f0
19/07:29:22|[c0391d90] [c023eb10] ip_rcv_finish+0x164/0x398
19/07:29:22|[c0391dc0] [c021f6c8] netif_receive_skb+0x26c/0x378
19/07:29:22|[c0391df0] [c0222518] process_backlog+0x88/0xd8
19/07:29:22|[c0391e20] [c02228a8] net_rx_action+0x8c/0x17c
19/07:29:22|[c0391e50] [c003aaa4] __do_softirq+0xbc/0x138
19/07:29:22|[c0391e90] [c0003c1c] do_softirq+0x74/0x7c
19/07:29:22|[c0391ea0] [c003a6b8] irq_exit+0x64/0x7c
19/07:29:22|[c0391eb0] [c000410c] do_IRQ+0x9c/0xb4
19/07:29:22|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
19/07:29:22|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
19/07:29:22|[c0391fb0] [c000172c] rest_init+0x70/0x84
19/07:29:22|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
19/07:29:22|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
19/07:29:22|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
19/07:32:23|
Found 0 errors checking kernel image
19/07:32:24|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
19/07:32:24|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
19/07:32:24|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
19/07:32:24|SPIconfig done now to set up the LMK3200 registers
19/07:32:24|LMK #0 : regInit[0]=0x80000000
19/07:32:24|LMK #0 : regInit[1]=0x10070600
19/07:32:24|LMK #0 : regInit[2]=0x60601
19/07:32:24|LMK #0 : regInit[3]=0x60602
19/07:32:25|LMK #0 : regInit[4]=0x60603
19/07:32:25|LMK #0 : regInit[5]=0x70624
19/07:32:25|LMK #0 : regInit[6]=0x70605
19/07:32:25|LMK #0 : regInit[7]=0x70606
19/07:32:25|LMK #0 : regInit[8]=0x70627
19/07:32:25|LMK #0 : regInit[9]=0x10000908
19/07:32:25|LMK #0 : regInit[10]=0xA0022A09
19/07:32:25|LMK #0 : regInit[11]=0x82800B
19/07:32:25|LMK #0 : regInit[12]=0x28C800D
19/07:32:25|LMK #0 : regInit[13]=0x830020E
19/07:32:25|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
19/07:32:25|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
19/07:32:25|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
19/07:32:25|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
19/07:32:26|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
19/07:32:29|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xcf 0xac
19/07:32:35|Using Xilinx Virtex440 machine description
19/07:32:36|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
19/07:32:36|Zone PFN ranges:
19/07:32:36|  DMA      0x00000000 -> 0x00007000
19/07:32:36|  Normal   0x00007000 -> 0x00007000
19/07:32:36|Movable zone start PFN for each node
19/07:32:36|early_node_map[1] active PFN ranges
19/07:32:36|    0: 0x00000000 -> 0x00007000
19/07:32:36|MMU: Allocated 1088 bytes of context maps for 255 contexts
19/07:32:36|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
19/07:32:36|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
19/07:32:36|PID hash table entries: 512 (order: 9, 2048 bytes)
19/07:32:36|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
19/07:32:37|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
19/07:32:37|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
19/07:32:37|Kernel virtual memory layout:
19/07:32:37|  * 0xffffe000..0xfffff000  : fixmap
19/07:32:37|  * 0xfde00000..0xfe000000  : consistent mem
19/07:32:37|  * 0xfde00000..0xfde00000  : early ioremap
19/07:32:37|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
19/07:32:37|NR_IRQS:512
19/07:32:37|clocksource: timebase mult[a00000] shift[22] registered
19/07:32:37|Console: colour dummy device 80x25
19/07:32:37|Mount-cache hash table entries: 512
19/07:32:37|NET: Registered protocol family 16
19/07:32:37|PCI: Probing PCI hardware
19/07:32:37|bio: create slab <bio-0> at 0
19/07:32:37|NET: Registered protocol family 2
19/07:32:37|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
19/07:32:37|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
19/07:32:37|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
19/07:32:37|TCP: Hash tables configured (established 4096 bind 4096)
19/07:32:37|TCP reno registered
19/07:32:37|NET: Registered protocol family 1
19/07:32:37|ROMFS MTD (C) 2007 Red Hat, Inc.
19/07:32:38|msgmni has been set to 214
19/07:32:38|io scheduler noop registered
19/07:32:38|io scheduler anticipatory registered
19/07:32:38|io scheduler deadline registered
19/07:32:38|io scheduler cfq registered (default)
19/07:32:38|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
19/07:32:38|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
19/07:32:38|console [ttyS0] enabled
19/07:32:38|brd: module loaded
19/07:32:38|loop: module loaded
19/07:32:38|Device Tree Probing 'ethernet'
19/07:32:38|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:cf:ac
19/07:32:38|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
19/07:32:38|XLlTemac: DCR address: 0x80
19/07:32:38|XLlTemac: buffer descriptor size: 32768 (0x8000)
19/07:32:38|XLlTemac: Allocating DMA descriptors with kmalloc
19/07:32:38|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
19/07:32:38|XTemac: PHY detected at address 7.
19/07:32:38|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
19/07:32:38|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39| Intel/Sharp Extended Query Table at 0x010A
19/07:32:39|Using buffer write method
19/07:32:39|cfi_cmdset_0001: Erase suspend on write enabled
19/07:32:39|cmdlinepart partition parsing not available
19/07:32:39|RedBoot partition parsing not available
19/07:32:39|Creating 5 MTD partitions on "fc000000.flash":
19/07:32:39|0x000000000000-0x000000500000 : "golden_firmware"
19/07:32:39|0x000000500000-0x000000800000 : "golden_kernel"
19/07:32:39|0x000000800000-0x000000d00000 : "user_firmware"
19/07:32:39|0x000000d00000-0x000000fe0000 : "user_kernel"
19/07:32:39|0x000000fe0000-0x000001000000 : "env_variables"
19/07:32:39|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
19/07:32:39|SPI: XIlinx spi: bus number now 32766
19/07:32:39|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
19/07:32:39|SPI: XIlinx spi: bus number now 32765
19/07:32:40|mice: PS/2 mouse device common for all mice
19/07:32:40|Device Tree Probing 'i2c'
19/07:32:40| #0 at 0x81600000 mapped to 0xD1030000, irq=22
19/07:32:40|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
19/07:32:40|TCP cubic registered
19/07:32:40|NET: Registered protocol family 17
19/07:32:40|RPC: Registered udp transport module.
19/07:32:40|RPC: Registered tcp transport module.
19/07:32:40|eth0: XLlTemac: Options: 0x3fa
19/07:32:40|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
19/07:32:40|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
19/07:32:40|eth0: XLlTemac: speed set to 1000Mb/s
19/07:32:43|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
19/07:32:43|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
19/07:32:43|Sending DHCP requests ., OK
19/07:32:44|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.12
19/07:32:44|IP-Config: Complete:
19/07:32:44|     device=eth0, addr=192.168.11.12, mask=255.255.255.0, gw=255.255.255.255,
19/07:32:44|     host=aida12, domain=dl.ac.uk, nis-domain=nuclear.physics,
19/07:32:44|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12
19/07:32:44|Looking up port of RPC 100003/2 on 192.168.11.99
19/07:32:44|Looking up port of RPC 100005/1 on 192.168.11.99
19/07:32:44|VFS: Mounted root (nfs filesystem) on device 0:12.
19/07:32:44|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
19/07:32:45|Starting sysinit...
19/07:32:45|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
19/07:32:46|            Press 'I' to enter interactive startup.
19/07:32:46|Setting clock  (utc): Mon Apr 19 07:32:47 BST 2021 [  OK  ]
19/07:32:47|Building the cache [  OK  ]
19/07:32:47|Setting hostname aida12:  [  OK  ]
19/07:32:48|Mounting local filesystems:  [  OK  ]
19/07:32:50|Enabling /etc/fstab swaps:  [  OK  ]
19/07:32:53|Finishing sysinit...
INIT: Entering runlevel: 3
19/07:32:56|Entering non-interactive startup
19/07:32:56|FATAL: Module ipv6 not found.
19/07:32:57|Bringing up loopback interface:  [  OK  ]
19/07:32:59|FATAL: Module ipv6 not found.
19/07:33:00|Starting system logger: [  OK  ]
19/07:33:01|Starting kernel logger: [  OK  ]
19/07:33:01|Starting rpcbind: [  OK  ]
19/07:33:03|Mounting NFS filesystems:  [  OK  ]
19/07:33:03|Mounting other filesystems:  [  OK  ]
19/07:33:04|Starting xinetd: [  OK  ]
19/07:33:05|Starting midas:  Starting MIDAS Data Acquisition for aida12
19/07:33:05|xaida: device parameters: base=0x81000000 size=0x200000
19/07:33:12|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
19/07:33:12|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
19/07:33:12|xaida: driver assigned major number 254
19/07:33:12|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
19/07:33:17|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
19/07:33:17|AIDAMEM: aidamem: driver assigned major number 253
19/07:33:17|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
19/07:33:24|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
19/07:33:24|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
19/07:33:24|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
19/07:33:24|Computer Name = aida12; Temp Directory = /tmp/tcl361
19/07:33:31|
19/07:33:34|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
  280   Sat Apr 24 15:28:17 2021 TDaida05 crash - 11:08:07 Tuesday 20 April
From ttyUSB7 system console log

20/09:59:30|get_ASICBlk : Blk=1 : bytes = 793600 : Blk_Status = 00000088 : Offset = 00100000 : Databuffer : 273090556 : 0 => C0017F1D : 1 => 0FAC4BAF
20/09:59:31|Last Data : Databuffer  273884152 : 0 => 80426EF4 : 1 => 0BD70000
20/09:59:31| C03E7F21 : 03FCCB77 | C03D7EF5 : 03FCEE9F | C0047DA3 : 03FCF5A7 | C0207E21 : 03FCF5A7
20/09:59:31| C0317E77 : 03FCF5A7 | C0087E6D : 03FCF66F | C0107E98 : 03FCF66F | C0217ED8 : 03FCF66F
20/09:59:31| C0327E19 : 03FCF66F | C00B7DF0 : 03FCF737 | C0117DD5 : 03FCF737 | C0227F27 : 03FCF737
20/09:59:31| C03C7F23 : 03FCF737 | C0137EF9 : 03FCF7FF | C0297EB7 : 03FCF7FF | C03E7EC7 : 03FCF7FF
20/09:59:31| C0167DFE : 03FCF8C7 | C02A7FC3 : 03FCF8C7 | C03F7E82 : 03FCF8C7 | C01B7E91 : 03FCF98F
20/09:59:31| C02F7E8F : 03FCF98F | C01D7DC4 : 03FCFA57 | C01E7D40 : 03FCFB1F | 80426EF3 : 03FD0000
20/09:59:32| 8050023F : 03FD0000 | C03D7F1A : 03FD303F | 80426EF3 : 03FD4000 | 8050023F : 03FD4000
20/09:59:32| C0317F2F : 03FD5047 | C0047DF2 : 03FD510F | C01B7EAD : 03FD54F7 | C03D7F09 : 03FD5D8F
20/09:59:32|SIGPIPE ERROR - Error in xfer ASIC procedures: location=0
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:37|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:39|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:40|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_bxaida: release:
20/10:00:41|loAIDAMEM: aidamem_release:
20/10:00:41|ck busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|xfer_block busy
20/10:00:41|SIGSEGV ERROR - Error in xfer ASIC procedures: location=0
20/10:00:41|xfer_block busy
20/10:00:41|do_GetState returned z=0 and 1
20/10:01:27|executing generic doStop
20/10:02:41|WriteRegister failed: Name=SigTask.EXEC.Usr1; Data=0; Code= POSIX ESRCH {no such process}; Info= sending signal SIGUSR1 to process 375 failed: no such 
process
20/10:02:42|StopC:
20/10:02:42|do_GetState returned z=0 and 6
20/10:02:47|do_GetState returned z=0 and 6
20/10:02:58|do_GetState returned z=0 and 6
20/10:03:10|do_GetState returned z=0 and 6
20/10:03:21|do_GetState returned z=0 and 6
20/10:03:33|do_GetState returned z=0 and 6
20/10:03:44|do_GetState returned z=0 and 6
20/10:03:56|do_GetState returned z=0 and 6
20/10:04:07|do_GetState returned z=0 and 6
20/10:04:19|do_GetState returned z=0 and 6
20/10:04:30|do_GetState returned z=0 and 6
20/10:04:42|do_GetState returned z=0 and 6
20/10:04:53|do_GetState returned z=0 and 6
20/10:05:04|do_GetState returned z=0 and 6
20/10:05:16|do_GetState returned z=0 and 6
20/10:05:27|do_GetState returned z=0 and 6
20/10:05:39|do_GetState returned z=0 and 6
20/10:05:50|do_GetState returned z=0 and 6
20/10:06:02|do_GetState returned z=0 and 6
20/10:06:12|executing generic doReset
20/10:06:21|WriteRegister failed: Name=SigTask.EXEC.Usr1; Data=0; Code= POSIX ESRCH {no such process}; Info= sending signal SIGUSR1 to process 375 failed: no such 
process
20/10:06:22|ResetC:
20/10:06:22|do_GetState returned z=0 and 6
20/10:06:22|do_GetState returned z=0 and 6
20/10:06:34|do_GetState returned z=0 and 6
20/10:06:45|do_GetState returned z=0 and 6
20/10:06:57|do_GetState returned z=0 and 6
INIT: Switching to runlevel: 6
INStarting killall:  Unmounting NFS filesystems:  umount: /MIDAS: not mounted
20/10:07:15|[  OK  ]
20/10:07:15|Stopping rpcbind: [  OK  ]
20/10:07:17|Shutting down kernel logger: [  OK  ]
20/10:07:19|Shutting down system logger: [  OK  ]
20/10:07:19|Stopping xinetd: [  OK  ]
20/10:07:19|[  OK  ]
20/10:07:19|Sending all processes the TERM signal...
20/10:07:20|xaida: release:
20/10:07:20|Sending all processes the KILL signal...
20/10:07:25|Saving random seed:
20/10:07:25|Syncing hardware clock to system time xaida: open:
20/10:07:25|xaida: release:
20/10:07:25|xaida: open:
20/10:07:25|xaida: ioctl: called with command 536899587
20/10:07:25|xaida: ioctl: called with command 1076129801
20/10:07:25|RTCxaida: release:
20/10:07:25|_RD_TIME: Inappropriate ioctl for device
20/10:07:25|ioctl() to /dev/rtc to read the time failed.
20/10:07:25|
20/10:07:25|Please stand by while rebooting the system...
Sj&#65533;5õÅ07:27|Restarting system.
20/10:07:29|ÉU&#65533;×®­Ë
                   R&#65533;   $T,W¤']Z&#65533;¥anuary 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/10:07:29|
Found 0 errors checking kernel image
20/10:07:30|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/10:07:30|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/10:07:31|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/10:07:31|SPIconfig done now to set up the LMK3200 registers
20/10:07:31|LMK #0 : regInit[0]=0x80000000
20/10:07:31|LMK #0 : regInit[1]=0x10070600
20/10:07:31|LMK #0 : regInit[2]=0x60601
20/10:07:31|LMK #0 : regInit[3]=0x60602
20/10:07:31|LMK #0 : regInit[4]=0x60603
20/10:07:31|LMK #0 : regInit[5]=0x70624
20/10:07:31|LMK #0 : regInit[6]=0x70605
20/10:07:31|LMK #0 : regInit[7]=0x70606
20/10:07:31|LMK #0 : regInit[8]=0x70627
20/10:07:31|LMK #0 : regInit[9]=0x10000908
20/10:07:31|LMK #0 : regInit[10]=0xA0022A09
20/10:07:31|LMK #0 : regInit[11]=0x82800B
20/10:07:31|LMK #0 : regInit[12]=0x28C800D
20/10:07:31|LMK #0 : regInit[13]=0x830020E
20/10:07:31|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/10:07:32|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/10:07:32|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/10:07:32|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/10:07:32|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/10:07:35|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xd7 0xcc
20/10:07:42|Using Xilinx Virtex440 machine description
20/10:07:42|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/10:07:42|Zone PFN ranges:
20/10:07:42|  DMA      0x00000000 -> 0x00007000
20/10:07:42|  Normal   0x00007000 -> 0x00007000
20/10:07:42|Movable zone start PFN for each node
20/10:07:42|early_node_map[1] active PFN ranges
20/10:07:43|    0: 0x00000000 -> 0x00007000
20/10:07:43|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/10:07:43|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/10:07:43|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/10:07:43|PID hash table entries: 512 (order: 9, 2048 bytes)
20/10:07:43|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/10:07:43|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/10:07:43|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/10:07:43|Kernel virtual memory layout:
20/10:07:43|  * 0xffffe000..0xfffff000  : fixmap
20/10:07:43|  * 0xfde00000..0xfe000000  : consistent mem
20/10:07:43|  * 0xfde00000..0xfde00000  : early ioremap
20/10:07:43|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/10:07:43|NR_IRQS:512
20/10:07:43|clocksource: timebase mult[a00000] shift[22] registered
20/10:07:43|Console: colour dummy device 80x25
20/10:07:43|Mount-cache hash table entries: 512
20/10:07:43|NET: Registered protocol family 16
20/10:07:43|PCI: Probing PCI hardware
20/10:07:43|bio: create slab <bio-0> at 0
20/10:07:44|NET: Registered protocol family 2
20/10:07:44|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/10:07:44|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/10:07:44|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/10:07:44|TCP: Hash tables configured (established 4096 bind 4096)
20/10:07:44|TCP reno registered
20/10:07:44|NET: Registered protocol family 1
20/10:07:44|ROMFS MTD (C) 2007 Red Hat, Inc.
20/10:07:44|msgmni has been set to 214
20/10:07:44|io scheduler noop registered
20/10:07:44|io scheduler anticipatory registered
20/10:07:44|io scheduler deadline registered
20/10:07:44|io scheduler cfq registered (default)
20/10:07:44|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/10:07:44|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/10:07:44|console [ttyS0] enabled
20/10:07:44|brd: module loaded
20/10:07:44|loop: module loaded
20/10:07:44|Device Tree Probing 'ethernet'
20/10:07:44|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:d7:cc
20/10:07:44|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/10:07:44|XLlTemac: DCR address: 0x80
20/10:07:44|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/10:07:45|XLlTemac: Allocating DMA descriptors with kmalloc
20/10:07:45|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/10:07:45|XTemac: PHY detected at address 7.
20/10:07:45|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/10:07:45|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45| Intel/Sharp Extended Query Table at 0x010A
20/10:07:45|Using buffer write method
20/10:07:45|cfi_cmdset_0001: Erase suspend on write enabled
20/10:07:45|cmdlinepart partition parsing not available
20/10:07:45|RedBoot partition parsing not available
20/10:07:45|Creating 5 MTD partitions on "fc000000.flash":
20/10:07:45|0x000000000000-0x000000500000 : "golden_firmware"
20/10:07:45|0x000000500000-0x000000800000 : "golden_kernel"
20/10:07:45|0x000000800000-0x000000d00000 : "user_firmware"
20/10:07:46|0x000000d00000-0x000000fe0000 : "user_kernel"
20/10:07:46|0x000000fe0000-0x000001000000 : "env_variables"
20/10:07:46|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/10:07:46|SPI: XIlinx spi: bus number now 32766
20/10:07:46|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/10:07:46|SPI: XIlinx spi: bus number now 32765
20/10:07:46|mice: PS/2 mouse device common for all mice
20/10:07:46|Device Tree Probing 'i2c'
20/10:07:46| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/10:07:46|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/10:07:46|TCP cubic registered
20/10:07:46|NET: Registered protocol family 17
20/10:07:46|RPC: Registered udp transport module.
20/10:07:46|RPC: Registered tcp transport module.
20/10:07:46|eth0: XLlTemac: Options: 0x3fa
20/10:07:47|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/10:07:47|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/10:07:47|eth0: XLlTemac: speed set to 1000Mb/s
20/10:07:49|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/10:07:49|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/10:07:49|Sending DHCP requests ., OK
20/10:07:50|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.5
20/10:07:50|IP-Config: Complete:
20/10:07:50|     device=eth0, addr=192.168.11.5, mask=255.255.255.0, gw=255.255.255.255,
20/10:07:50|     host=aida05, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/10:07:50|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05
20/10:07:50|Looking up port of RPC 100003/2 on 192.168.11.99
20/10:07:51|Looking up port of RPC 100005/1 on 192.168.11.99
20/10:07:51|VFS: Mounted root (nfs filesystem) on device 0:12.
20/10:07:51|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/10:07:51|Starting sysinit...
20/10:07:51|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/10:07:52|            Press 'I' to enter interactive startup.
20/10:07:52|Setting clock  (utc): Tue Apr 20 10:07:52 BST 2021 [  OK  ]
20/10:07:53|Building the cache [  OK  ]
20/10:07:53|Setting hostname aida05:  [  OK  ]
20/10:07:54|Mounting local filesystems:  [  OK  ]
20/10:07:55|Enabling /etc/fstab swaps:  [  OK  ]
20/10:07:57|Finishing sysinit...
INIT: Entering runlevel: 3
20/10:08:00|Entering non-interactive startup
20/10:08:01|FATAL: Module ipv6 not found.
20/10:08:02|Bringing up loopback interface:  [  OK  ]
20/10:08:04|FATAL: Module ipv6 not found.
20/10:08:04|Starting system logger: [  OK  ]
20/10:08:05|Starting kernel logger: [  OK  ]
20/10:08:05|Starting rpcbind: [  OK  ]
20/10:08:06|Mounting NFS filesystems:  [  OK  ]
20/10:08:07|Mounting other filesystems:  [  OK  ]
20/10:08:07|Starting xinetd: [  OK  ]
20/10:08:08|Starting midas:  Starting MIDAS Data Acquisition for aida05
20/10:08:08|xaida: device parameters: base=0x81000000 size=0x200000
20/10:08:14|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/10:08:14|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/10:08:14|xaida: driver assigned major number 254
20/10:08:14|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/10:08:20|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/10:08:20|AIDAMEM: aidamem: driver assigned major number 253
20/10:08:20|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/10:08:26|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/10:08:27|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/10:08:27|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/10:08:27|Computer Name = aida05; Temp Directory = /tmp/tcl356
20/10:08:30|package limit is not available: can't find package limit
20/10:08:33|Running with default file descriptor limit
20/10:08:33|package setuid is not available: can't find package setuid
20/10:08:35|Running as user 0 group 0
20/10:08:35|
20/10:08:36|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
  279   Sat Apr 24 15:13:33 2021 TDaida09 crash - 08:10:52 Tuesday 20 April
From ttyUSB6 system console log

20/07:07:10|kernel BUG at mm/slab.c:2974!
20/07:07:10|Oops: Exception in kernel mode, sig: 5 [#1]
20/07:07:10|PREEMPT Xilinx Virtex440
20/07:07:10|Modules linked in: aidamem xdriver xh_spidev_register
20/07:07:10|NIP: c009211c LR: c00920b0 CTR: c0038e70
20/07:07:10|REGS: c0391bf0 TRAP: 0700   Not tainted  (2.6.31)
20/07:07:10|MSR: 00021000 <ME,CE>  CR: 24022048  XER: 00000000
20/07:07:10|TASK = c036e318[0] 'swapper' THREAD: c0390000
20/07:07:10|GPR00: 00000001 c0391ca0 c036e318 c680daf0 00000020 00000001 c6940000 00000000
20/07:07:11|GPR08: fffffff4 00010003 c680e400 c0390000 24008042 00005aa8 c03a0000 00000020
20/07:07:11|GPR16: c0390000 c03a069c c03a0000 c038c384 c038cc18 00000020 00000000 00200200
20/07:07:11|GPR24: 00100100 c0390000 00000000 c680dae8 c680dae0 c680ae00 00000010 c680e400
20/07:07:11|NIP [c009211c] cache_alloc_refill+0x130/0x608
20/07:07:11|LR [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:11|Call Trace:
20/07:07:11|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608 (unreliable)
20/07:07:11|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:11|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:11|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:11|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:11|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:11|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:11|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:11|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:11|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:11|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:11|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:12|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:12|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:12|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:12|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:12|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:12|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:12|Instruction dump:
20/07:07:12|2f1e0000 409900f4 387c0010 3b7c0008 80dc0000 7f9c3000 419e014c 81060010
20/07:07:12|801d001c 7c004010 38000000 7c000114 <0f000000> 81260010 801d001c 7f804840
20/07:07:12|Kernel panic - not syncing: Fatal exception in interrupt
20/07:07:12|Call Trace:
20/07:07:12|[c0391a40] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/07:07:12|[c0391a80] [c00345bc] panic+0x94/0x168
20/07:07:12|[c0391ad0] [c000bd44] die+0x178/0x18c
20/07:07:12|[c0391af0] [c000c000] _exception+0x164/0x1b4
20/07:07:12|[c0391be0] [c000e978] ret_from_except_full+0x0/0x4c
20/07:07:12|[c0391ca0] [c00920b0] cache_alloc_refill+0xc4/0x608
20/07:07:12|[c0391d00] [c00927d8] kmem_cache_alloc+0xc4/0xcc
20/07:07:12|[c0391d20] [c0042420] __sigqueue_alloc+0x50/0xb8
20/07:07:12|[c0391d40] [c0042938] __send_signal+0x78/0x260
20/07:07:13|[c0391d70] [c0042f78] group_send_sig_info+0x70/0x9c
20/07:07:13|[c0391da0] [c00438a8] kill_pid_info+0x48/0x8c
20/07:07:13|[c0391dc0] [c0038e8c] it_real_fn+0x1c/0x30
20/07:07:13|[c0391dd0] [c0050c40] hrtimer_run_queues+0x184/0x240
20/07:07:13|[c0391e30] [c0040ba8] run_local_timers+0x10/0x2c
20/07:07:13|[c0391e40] [c0040bf4] update_process_times+0x30/0x70
20/07:07:13|[c0391e60] [c005a000] tick_periodic+0x34/0xe8
20/07:07:13|[c0391e70] [c005a0d4] tick_handle_periodic+0x20/0x120
20/07:07:13|[c0391eb0] [c000af70] timer_interrupt+0xa4/0x10c
20/07:07:13|[c0391ed0] [c000e9c4] ret_from_except+0x0/0x18
20/07:07:13|[c0391f90] [c0006fac] cpu_idle+0xcc/0xdc
20/07:07:13|[c0391fb0] [c000172c] rest_init+0x70/0x84
20/07:07:13|[c0391fc0] [c0341854] start_kernel+0x230/0x2ac
20/07:07:13|[c0391ff0] [c0000204] skpinv+0x194/0x1d0
20/07:07:13|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/07:10:14|
Found 0 errors checking kernel image
20/07:10:15|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/07:10:15|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/07:10:15|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/07:10:15|SPIconfig done now to set up the LMK3200 registers
20/07:10:15|LMK #0 : regInit[0]=0x80000000
20/07:10:15|LMK #0 : regInit[1]=0x10070600
20/07:10:15|LMK #0 : regInit[2]=0x60601
20/07:10:15|LMK #0 : regInit[3]=0x60602
20/07:10:16|LMK #0 : regInit[4]=0x60603
20/07:10:16|LMK #0 : regInit[5]=0x70624
20/07:10:16|LMK #0 : regInit[6]=0x70605
20/07:10:16|LMK #0 : regInit[7]=0x70606
20/07:10:16|LMK #0 : regInit[8]=0x70627
20/07:10:16|LMK #0 : regInit[9]=0x10000908
20/07:10:16|LMK #0 : regInit[10]=0xA0022A09
20/07:10:16|LMK #0 : regInit[11]=0x82800B
20/07:10:16|LMK #0 : regInit[12]=0x28C800D
20/07:10:16|LMK #0 : regInit[13]=0x830020E
20/07:10:16|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 943 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/07:10:16|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/07:10:16|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/07:10:16|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
Cal DCMs not locked
ADC calibrate failed
Jumping to kernel simpleboot...
20/07:10:17|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/07:10:20|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xee
20/07:10:26|Using Xilinx Virtex440 machine description
20/07:10:27|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/07:10:27|Zone PFN ranges:
20/07:10:27|  DMA      0x00000000 -> 0x00007000
20/07:10:27|  Normal   0x00007000 -> 0x00007000
20/07:10:27|Movable zone start PFN for each node
20/07:10:27|early_node_map[1] active PFN ranges
20/07:10:27|    0: 0x00000000 -> 0x00007000
20/07:10:27|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/07:10:27|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/07:10:27|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/07:10:27|PID hash table entries: 512 (order: 9, 2048 bytes)
20/07:10:27|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/07:10:27|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/07:10:28|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/07:10:28|Kernel virtual memory layout:
20/07:10:28|  * 0xffffe000..0xfffff000  : fixmap
20/07:10:28|  * 0xfde00000..0xfe000000  : consistent mem
20/07:10:28|  * 0xfde00000..0xfde00000  : early ioremap
20/07:10:28|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/07:10:28|NR_IRQS:512
20/07:10:28|clocksource: timebase mult[a00000] shift[22] registered
20/07:10:28|Console: colour dummy device 80x25
20/07:10:28|Mount-cache hash table entries: 512
20/07:10:28|NET: Registered protocol family 16
20/07:10:28|PCI: Probing PCI hardware
20/07:10:28|bio: create slab <bio-0> at 0
20/07:10:28|NET: Registered protocol family 2
20/07:10:28|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/07:10:28|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/07:10:28|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/07:10:28|TCP: Hash tables configured (established 4096 bind 4096)
20/07:10:28|TCP reno registered
20/07:10:28|NET: Registered protocol family 1
20/07:10:28|ROMFS MTD (C) 2007 Red Hat, Inc.
20/07:10:28|msgmni has been set to 214
20/07:10:29|io scheduler noop registered
20/07:10:29|io scheduler anticipatory registered
20/07:10:29|io scheduler deadline registered
20/07:10:29|io scheduler cfq registered (default)
20/07:10:29|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/07:10:29|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/07:10:29|console [ttyS0] enabled
20/07:10:29|brd: module loaded
20/07:10:29|loop: module loaded
20/07:10:29|Device Tree Probing 'ethernet'
20/07:10:29|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:ee
20/07:10:29|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/07:10:29|XLlTemac: DCR address: 0x80
20/07:10:29|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/07:10:29|XLlTemac: Allocating DMA descriptors with kmalloc
20/07:10:29|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/07:10:29|XTemac: PHY detected at address 7.
20/07:10:29|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/07:10:29|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/07:10:29| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30| Intel/Sharp Extended Query Table at 0x010A
20/07:10:30|Using buffer write method
20/07:10:30|cfi_cmdset_0001: Erase suspend on write enabled
20/07:10:30|cmdlinepart partition parsing not available
20/07:10:30|RedBoot partition parsing not available
20/07:10:30|Creating 5 MTD partitions on "fc000000.flash":
20/07:10:30|0x000000000000-0x000000500000 : "golden_firmware"
20/07:10:30|0x000000500000-0x000000800000 : "golden_kernel"
20/07:10:30|0x000000800000-0x000000d00000 : "user_firmware"
20/07:10:30|0x000000d00000-0x000000fe0000 : "user_kernel"
20/07:10:30|0x000000fe0000-0x000001000000 : "env_variables"
20/07:10:30|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/07:10:30|SPI: XIlinx spi: bus number now 32766
20/07:10:30|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/07:10:30|SPI: XIlinx spi: bus number now 32765
20/07:10:30|mice: PS/2 mouse device common for all mice
20/07:10:31|Device Tree Probing 'i2c'
20/07:10:31| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/07:10:31|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/07:10:31|TCP cubic registered
20/07:10:31|NET: Registered protocol family 17
20/07:10:31|RPC: Registered udp transport module.
20/07:10:31|RPC: Registered tcp transport module.
20/07:10:31|eth0: XLlTemac: Options: 0x3fa
20/07:10:31|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/07:10:31|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/07:10:31|eth0: XLlTemac: speed set to 1000Mb/s
20/07:10:33|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/07:10:34|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/07:10:34|Sending DHCP requests ., OK
20/07:10:35|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.9
20/07:10:35|IP-Config: Complete:
20/07:10:35|     device=eth0, addr=192.168.11.9, mask=255.255.255.0, gw=255.255.255.255,
20/07:10:35|     host=aida09, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/07:10:35|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09
20/07:10:35|Looking up port of RPC 100003/2 on 192.168.11.99
20/07:10:35|Looking up port of RPC 100005/1 on 192.168.11.99
20/07:10:35|VFS: Mounted root (nfs filesystem) on device 0:12.
20/07:10:35|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/07:10:36|Starting sysinit...
20/07:10:36|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/07:10:37|            Press 'I' to enter interactive startup.
20/07:10:37|Setting clock  (utc): Tue Apr 20 07:10:37 BST 2021 [  OK  ]
20/07:10:38|Building the cache [  OK  ]
20/07:10:38|Setting hostname aida09:  [  OK  ]
20/07:10:39|Mounting local filesystems:  [  OK  ]
20/07:10:40|Enabling /etc/fstab swaps:  [  OK  ]
20/07:10:43|Finishing sysinit...
INIT: Entering runlevel: 3
20/07:10:46|Entering non-interactive startup
20/07:10:46|FATAL: Module ipv6 not found.
20/07:10:47|Bringing up loopback interface:  [  OK  ]
20/07:10:49|FATAL: Module ipv6 not found.
20/07:10:50|Starting system logger: [  OK  ]
20/07:10:50|Starting kernel logger: [  OK  ]
20/07:10:50|Starting rpcbind: [  OK  ]
20/07:10:52|Mounting NFS filesystems:  [  OK  ]
20/07:10:53|Mounting other filesystems:  [  OK  ]
20/07:10:53|Starting xinetd: [  OK  ]
20/07:10:54|Starting midas:  Starting MIDAS Data Acquisition for aida09
20/07:10:54|xaida: device parameters: base=0x81000000 size=0x200000
20/07:11:00|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/07:11:00|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/07:11:00|xaida: driver assigned major number 254
20/07:11:00|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/07:11:06|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/07:11:06|AIDAMEM: aidamem: driver assigned major number 253
20/07:11:06|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/07:11:12|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/07:11:12|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/07:11:12|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/07:11:13|Computer Name = aida09; Temp Directory = /tmp/tcl361
20/07:11:17|package limit is not available: can't find package limit
20/07:11:20|Running with default file descriptor limit
20/07:11:20|package setuid is not available: can't find package setuid
20/07:11:21|Running as user 0 group 0
20/07:11:21|
20/07:11:22|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
20/07:11:22|
20/07:11:22|Built without pthreads
20/07:11:22|xaida: open:
20/07:11:22|
20/07:11:22|Creating NetAIDAMEM: aidamem_open:
20/07:11:22|Vars
20/07:11:22|Output buffer length = 65504; format option = 4; transfer option = 3
20/07:11:22|EB transfer option = 3
20/07:11:22|NetVars created and initialised
20/07:11:22|Statistics thread starting
20/07:11:22|Statistics thread created
20/07:11:22|Stat/Rate creation thread starting
20/07:11:22|Data Acquisition task has PID 375
20/07:11:22|Stat/Rate creation thread created
20/07:11:22|Hit/Rate creation thread starting
20/07:11:22|Hit/Rate creation thread created
20/07:11:22|AIDA Heartbeat thread starting
20/07:11:22|Heartbeat thread created
20/07:11:22|Installing signal handlers
20/07:11:22|Done
20/07:11:22|ModuleNum = 0
20/07:11:22|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 03350706
20/07:11:22|Spectra table initialised
20/07:11:22|AIDA Data Acquisition now all ready to start
20/07:11:23|SIGBUS, SIGSEGV and SIGPIPE traps setup
20/07:11:23|/debug user "debug" password "5tvq-4iyxl 4"
20/07:11:30|httpd started on port 8015
20/07:11:30|
20/07:11:30|Cannot use /MIDAS/config/TclHttpd/aida09@8015/startup.tcl
20/07:11:30|Custom startup from /MIDAS/config/TclHttpd/aida09/startup.tcl
20/07:11:30|XAIDA Access package 1.0
20/07:11:32|/XAIDAAccessServer
20/07:11:32|XAD9252 Access package 1.0
20/07:11:32|/XAD9252AccessServer
20/07:11:32|/DataBaseAccessServer
20/07:11:32|[  OK  ]
20/07:11:32|/NetVarService
20/07:11:32|/SigTaskService
20/07:11:32|Loaded MemSasAccess
20/07:11:33|/SpectrumService
20/07:11:33|loading tcl/AIDARunControl.tcl for namespace ::
20/07:11:33|/DataAcquisitionControlServer
20/07:11:33|DefineMessage unknown
20/07:11:33|
20/07:11:33|Run Control Server Implementation for AIDA
20/07:11:33|DENX ELDK version 4.2 build 2008-04-01
20/07:11:33|Linux 2.6.31 on a ppc
20/07:11:33|
20/07:11:33|aida09 login: RunControlServer loaded
20/07:11:33|loading Html/RunControl/implementation.tcl
20/07:11:33|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
20/07:11:34|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
20/07:11:34|Created UI registers
20/07:11:34|RunControl loaded
20/07:11:34|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
20/07:11:34|AIDA RunControl loaded
20/07:11:35|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
20/07:11:35|do_GetState returned z=0 and 8
20/07:12:03|do_GetState returned z=0 and 8
20/07:12:09|do_GetState returned z=0 and 8
20/07:12:21|xaida: open:
20/07:12:35|do_GetState returned z=0 and 8
20/07:12:36|
  278   Sat Apr 24 13:33:44 2021 TDaida01 restart - 22.07UTC+2 Tuesday 20 April
From aida01 /var/log/messages

Apr 20 21:04:06 aida01 kernel: BUG: scheduling while atomic: tclsh8.5/398/0x00000201
Apr 20 21:04:08 aida01 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 20 21:04:09 aida01 kernel: Call Trace:
Apr 20 21:04:11 aida01 kernel: [c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
Apr 20 21:04:12 aida01 kernel: [c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
Apr 20 21:04:14 aida01 kernel: [c6941aa0] [c02a8020] schedule+0x2f4/0x34c
Apr 20 21:07:53 aida01 syslogd 1.4.2: restart.

From ttyUSB0 system console log

20/21:04:06|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:06|Call Trace:
20/21:04:06|[c6941a50] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:06|[c6941a90] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:07|[c6941aa0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:07|[c6941ae0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:07|[c6941af0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:07|[c6941b10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:07|[c6941b50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:07|[c6941b90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:07|[c6941ba0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:07|[c6941bd0] [c013da58] nfs3_rpc_wrapper+0x48/0x84
20/21:04:07|[c6941bf0] [c013e734] nfs3_proc_access+0xcc/0x1ac
20/21:04:07|[c6941cc0] [c0129644] nfs_do_access+0xb8/0x438
20/21:04:07|[c6941d00] [c0129a6c] nfs_permission+0xa8/0x1c4
20/21:04:07|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:07|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:07|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:07|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:07|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:07|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:07|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:07|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:07|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:08|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:08|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:08|Call Trace:
20/21:04:08|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:08|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:08|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:08|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:08|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:08|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:08|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:08|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:08|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:08|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:08|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:08|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:08|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:08|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:09|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:09|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:09|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:09|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:09|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:09|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:09|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:09|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:09|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:09|Call Trace:
20/21:04:09|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:09|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:09|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:09|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:09|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:09|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:09|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:09|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:09|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:09|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:10|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:10|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:10|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:10|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:10|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:10|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:10|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:10|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:10|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:10|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:10|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:10|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:10|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:10|Call Trace:
20/21:04:10|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:10|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:10|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:10|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:10|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:10|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:11|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:11|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:11|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:11|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:11|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:11|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:11|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:11|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:11|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:11|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:11|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:11|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:11|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:11|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:11|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:11|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:11|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:11|Call Trace:
20/21:04:11|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:11|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:12|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:12|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:12|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:12|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:12|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:12|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:12|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:12|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:12|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:12|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:12|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:12|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:12|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:12|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:12|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:12|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:12|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:12|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:12|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:12|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:13|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:13|Call Trace:
20/21:04:13|[c6941a90] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:13|[c6941ad0] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:13|[c6941ae0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:13|[c6941b20] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:13|[c6941b30] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:13|[c6941b50] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:13|[c6941b90] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:13|[c6941bd0] [c028d120] rpc_run_task+0x44/0x90
20/21:04:13|[c6941be0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:13|[c6941c10] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:13|[c6941c40] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:13|[c6941d00] [c0129b3c] nfs_permission+0x178/0x1c4
20/21:04:13|[c6941d20] [c00a0bf4] inode_permission+0x94/0xbc
20/21:04:13|[c6941d30] [c00a2bc0] __link_path_walk+0xb0/0xf7c
20/21:04:13|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:13|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:13|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:13|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:14|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:14|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:14|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:14|BUG: scheduling while atomic: tclsh8.5/398/0x00000201
20/21:04:14|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:14|Call Trace:
20/21:04:14|[c6941950] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:14|[c6941990] [c002d41c] __schedule_bug+0x68/0x7c
20/21:04:14|[c69419a0] [c02a8020] schedule+0x2f4/0x34c
20/21:04:14|[c69419e0] [c02946e4] rpc_wait_bit_killable+0x24/0x4c
20/21:04:14|[c69419f0] [c02a8a04] __wait_on_bit+0x88/0xe0
20/21:04:14|[c6941a10] [c02a8abc] out_of_line_wait_on_bit+0x60/0x74
20/21:04:14|[c6941a50] [c0294e78] __rpc_execute+0x134/0x300
20/21:04:14|[c6941a90] [c028d120] rpc_run_task+0x44/0x90
20/21:04:14|[c6941aa0] [c028d234] rpc_call_sync+0x54/0x84
20/21:04:14|[c6941ad0] [c0136ee4] nfs_proc_getattr+0x5c/0xa8
20/21:04:14|[c6941b00] [c012ee68] __nfs_revalidate_inode+0xd0/0x1f0
20/21:04:14|[c6941bc0] [c012bd70] nfs_lookup_revalidate+0x3c0/0x464
20/21:04:14|[c6941d00] [c00a0f7c] do_lookup+0x54/0x20c
20/21:04:14|[c6941d30] [c00a308c] __link_path_walk+0x57c/0xf7c
20/21:04:15|[c6941da0] [c00a3ad4] path_walk+0x48/0xc8
20/21:04:15|[c6941dd0] [c00a3c44] do_path_lookup+0x64/0x6c
20/21:04:15|[c6941df0] [c00a4b14] do_filp_open+0xc8/0x914
20/21:04:15|[c6941ec0] [c009d990] open_exec+0x2c/0x100
20/21:04:15|[c6941ee0] [c009dffc] do_execve+0xb0/0x270
20/21:04:15|[c6941f20] [c00068e8] sys_execve+0x50/0x7c
20/21:04:15|[c6941f40] [c000e364] ret_from_syscall+0x0/0x3c
20/21:04:15|Page fault in user mode with in_atomic() = 1 mm = c6178c80
20/21:04:15|NIP = fb85a30  MSR = 2d000
20/21:04:15|Oops: Weird page fault, sig: 11 [#1]
20/21:04:15|PREEMPT Xilinx Virtex440
20/21:04:15|Modules linked in: aidamem xdriver xh_spidev_register
20/21:04:15|NIP: 0fb85a30 LR: 0ff85338 CTR: c01288f4
20/21:04:15|REGS: c6941f50 TRAP: 0401   Not tainted  (2.6.31)
20/21:04:15|MSR: 0002d000 <EE,PR,ME,CE>  CR: 44884044  XER: 00000000
20/21:04:15|TASK = c608e0c0[398] 'tclsh8.5' THREAD: c6940000
20/21:04:15|GPR00: 0ff85334 bf9a3870 4802b0a0 ffffffff 0fc094e0 54884044 fffffff2 0fc08d3c
20/21:04:15|GPR08: 0002d000 480240a8 00000000 bf9a3870 c6940000
20/21:04:15|NIP [0fb85a30] 0xfb85a30
20/21:04:15|LR [0ff85338] 0xff85338
20/21:04:16|Call Trace:
20/21:04:16|Kernel panic - not syncing: Fatal exception in interrupt
20/21:04:16|Call Trace:
20/21:04:16|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
20/21:04:16|[c6941e20] [c00345bc] panic+0x94/0x168
20/21:04:16|[c6941e70] [c000bd44] die+0x178/0x18c
20/21:04:16|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
20/21:04:16|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
20/21:04:16|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM
20/21:07:16|
Found 0 errors checking kernel image
20/21:07:18|VHDL version number 0X03350706
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup
20/21:07:18|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
20/21:07:18|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
20/21:07:18|SPIconfig done now to set up the LMK3200 registers
20/21:07:18|LMK #0 : regInit[0]=0x80000000
20/21:07:18|LMK #0 : regInit[1]=0x10070600
20/21:07:18|LMK #0 : regInit[2]=0x60601
20/21:07:18|LMK #0 : regInit[3]=0x60602
20/21:07:18|LMK #0 : regInit[4]=0x60603
20/21:07:18|LMK #0 : regInit[5]=0x70624
20/21:07:18|LMK #0 : regInit[6]=0x70605
20/21:07:18|LMK #0 : regInit[7]=0x70606
20/21:07:18|LMK #0 : regInit[8]=0x70627
20/21:07:18|LMK #0 : regInit[9]=0x10000908
20/21:07:18|LMK #0 : regInit[10]=0xA0022A09
20/21:07:18|LMK #0 : regInit[11]=0x82800B
20/21:07:18|LMK #0 : regInit[12]=0x28C800D
20/21:07:18|LMK #0 : regInit[13]=0x830020E
20/21:07:18|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 942 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
20/21:07:19|
Finished Clock setup LMK03200
completed LMK 3200 setup
Loaded all four ASICs with default settings
Setting the ADCs into calibration mode
20/21:07:19|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised
ADCs calibrated
20/21:07:19|
Control 32(0x81400400)=0x186
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
20/21:07:19|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
20/21:07:19|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
20/21:07:23|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xba 0x8a
20/21:07:29|Using Xilinx Virtex440 machine description
20/21:07:30|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
20/21:07:30|Zone PFN ranges:
20/21:07:30|  DMA      0x00000000 -> 0x00007000
20/21:07:30|  Normal   0x00007000 -> 0x00007000
20/21:07:30|Movable zone start PFN for each node
20/21:07:30|early_node_map[1] active PFN ranges
20/21:07:30|    0: 0x00000000 -> 0x00007000
20/21:07:30|MMU: Allocated 1088 bytes of context maps for 255 contexts
20/21:07:30|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
20/21:07:30|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
20/21:07:30|PID hash table entries: 512 (order: 9, 2048 bytes)
20/21:07:30|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
20/21:07:30|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
20/21:07:30|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
20/21:07:30|Kernel virtual memory layout:
20/21:07:30|  * 0xffffe000..0xfffff000  : fixmap
20/21:07:31|  * 0xfde00000..0xfe000000  : consistent mem
20/21:07:31|  * 0xfde00000..0xfde00000  : early ioremap
20/21:07:31|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
20/21:07:31|NR_IRQS:512
20/21:07:31|clocksource: timebase mult[a00000] shift[22] registered
20/21:07:31|Console: colour dummy device 80x25
20/21:07:31|Mount-cache hash table entries: 512
20/21:07:31|NET: Registered protocol family 16
20/21:07:31|PCI: Probing PCI hardware
20/21:07:31|bio: create slab <bio-0> at 0
20/21:07:31|NET: Registered protocol family 2
20/21:07:31|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
20/21:07:31|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
20/21:07:31|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
20/21:07:31|TCP: Hash tables configured (established 4096 bind 4096)
20/21:07:31|TCP reno registered
20/21:07:31|NET: Registered protocol family 1
20/21:07:31|ROMFS MTD (C) 2007 Red Hat, Inc.
20/21:07:31|msgmni has been set to 214
20/21:07:31|io scheduler noop registered
20/21:07:31|io scheduler anticipatory registered
20/21:07:31|io scheduler deadline registered
20/21:07:31|io scheduler cfq registered (default)
20/21:07:31|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
20/21:07:32|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
20/21:07:32|console [ttyS0] enabled
20/21:07:32|brd: module loaded
20/21:07:32|loop: module loaded
20/21:07:32|Device Tree Probing 'ethernet'
20/21:07:32|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:ba:8a
20/21:07:32|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
20/21:07:32|XLlTemac: DCR address: 0x80
20/21:07:32|XLlTemac: buffer descriptor size: 32768 (0x8000)
20/21:07:32|XLlTemac: Allocating DMA descriptors with kmalloc
20/21:07:32|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
20/21:07:32|XTemac: PHY detected at address 7.
20/21:07:32|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
20/21:07:32|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:32| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33| Intel/Sharp Extended Query Table at 0x010A
20/21:07:33|Using buffer write method
20/21:07:33|cfi_cmdset_0001: Erase suspend on write enabled
20/21:07:33|cmdlinepart partition parsing not available
20/21:07:33|RedBoot partition parsing not available
20/21:07:33|Creating 5 MTD partitions on "fc000000.flash":
20/21:07:33|0x000000000000-0x000000500000 : "golden_firmware"
20/21:07:33|0x000000500000-0x000000800000 : "golden_kernel"
20/21:07:33|0x000000800000-0x000000d00000 : "user_firmware"
20/21:07:33|0x000000d00000-0x000000fe0000 : "user_kernel"
20/21:07:33|0x000000fe0000-0x000001000000 : "env_variables"
20/21:07:33|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
20/21:07:33|SPI: XIlinx spi: bus number now 32766
20/21:07:33|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
20/21:07:33|SPI: XIlinx spi: bus number now 32765
20/21:07:33|mice: PS/2 mouse device common for all mice
20/21:07:33|Device Tree Probing 'i2c'
20/21:07:33| #0 at 0x81600000 mapped to 0xD1030000, irq=22
20/21:07:33|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
20/21:07:33|TCP cubic registered
20/21:07:33|NET: Registered protocol family 17
20/21:07:34|RPC: Registered udp transport module.
20/21:07:34|RPC: Registered tcp transport module.
20/21:07:34|eth0: XLlTemac: Options: 0x3fa
20/21:07:34|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
20/21:07:34|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
20/21:07:34|eth0: XLlTemac: speed set to 1000Mb/s
20/21:07:36|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
20/21:07:36|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
20/21:07:36|Sending DHCP requests ., OK
20/21:07:38|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.1
20/21:07:38|IP-Config: Complete:
20/21:07:38|     device=eth0, addr=192.168.11.1, mask=255.255.255.0, gw=255.255.255.255,
20/21:07:38|     host=aida01, domain=dl.ac.uk, nis-domain=nuclear.physics,
20/21:07:38|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01
20/21:07:38|Looking up port of RPC 100003/2 on 192.168.11.99
20/21:07:38|Looking up port of RPC 100005/1 on 192.168.11.99
20/21:07:38|VFS: Mounted root (nfs filesystem) on device 0:12.
20/21:07:38|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
20/21:07:39|Starting sysinit...
20/21:07:39|            Welcome to DENX & STFC Daresbury Embedded Linux Environment
20/21:07:40|            Press 'I' to enter interactive startup.
20/21:07:40|Setting clock  (utc): Tue Apr 20 21:07:40 BST 2021 [  OK  ]
20/21:07:40|Building the cache [  OK  ]
20/21:07:40|Setting hostname aida01:  [  OK  ]
20/21:07:42|Mounting local filesystems:  [  OK  ]
20/21:07:43|Enabling /etc/fstab swaps:  [  OK  ]
20/21:07:46|Finishing sysinit...
INIT: Entering runlevel: 3
20/21:07:49|Entering non-interactive startup
20/21:07:49|FATAL: Module ipv6 not found.
20/21:07:51|Bringing up loopback interface:  [  OK  ]
20/21:07:53|FATAL: Module ipv6 not found.
20/21:07:53|Starting system logger: [  OK  ]
20/21:07:54|Starting kernel logger: [  OK  ]
20/21:07:54|Starting rpcbind: [  OK  ]
20/21:07:56|Mounting NFS filesystems:  [  OK  ]
20/21:07:57|Mounting other filesystems:  [  OK  ]
20/21:07:57|Starting xinetd: [  OK  ]
20/21:07:58|Starting midas:  Starting MIDAS Data Acquisition for aida01
20/21:07:58|xaida: device parameters: base=0x81000000 size=0x200000
20/21:08:04|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
20/21:08:04|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
20/21:08:05|xaida: driver assigned major number 254
20/21:08:05|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
20/21:08:10|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
20/21:08:10|AIDAMEM: aidamem: driver assigned major number 253
20/21:08:10|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
20/21:08:17|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
20/21:08:17|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
20/21:08:17|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
20/21:08:17|Computer Name = aida01; Temp Directory = /tmp/tcl361
20/21:08:22|package limit is not available: can't find package limit
20/21:08:25|Running with default file descriptor limit
20/21:08:25|
20/21:08:26|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
  277   Sat Apr 24 13:25:23 2021 NH, TDAIDA FEE64 crash log for S460
https://docs.google.com/spreadsheets/d/1CyL5L-WLsh9fsS7CN989GC3ktqiPQOkG8YIDJrrGxYA/edit#gid=0
  276   Sat Apr 24 10:21:06 2021 NHSaturday 24.04.2021
I have restarted some more of the consoles that disappeared since the aidas-gsi issue last night
Grafana is working again

Run R52 CLOSED
Switched to no storage mode (NOTAPE/R1)

System Wide Checks
Clocks: OK
FADC: OK
WR: 
		 Base 		Current 	Difference
aida05 fault 	 0x31a2 : 	 0x31a2 : 	 0  
aida05 : WR status 0x10
 aida06 fault 	 0x22e7 : 	 0x22e8 : 	 1  
White Rabbit error counter test result: Passed 10, Failed 2
FPGA: OK
Memory: OK

aida05 ASIC maybe didn't complete setup properly (WR 0x10?)

Timestamps: OK (1)
Temps: OK

Will start gzipping S460 runs for migration to lustre & tape
Attachment 1: AnyDesk_2021-04-24_11-27-49.png
AnyDesk_2021-04-24_11-27-49.png
Attachment 2: AnyDesk_2021-04-24_11-28-50.png
AnyDesk_2021-04-24_11-28-50.png
  275   Sat Apr 24 05:17:51 2021 MLEnd of beam

Beam has now stopped (end of expriment) and background measurements starts.

  274   Sat Apr 24 04:50:01 2021 MLSpectra

Some screenshot of spectra attached

Attachment 1: 24-04-2021_540am_SpecLayoutId3.png
24-04-2021_540am_SpecLayoutId3.png
Attachment 2: 24-04-2021_540am_SpecLayId4.png
24-04-2021_540am_SpecLayId4.png
Attachment 3: 24-04-2021_540am_SpecLayId5.png
24-04-2021_540am_SpecLayId5.png
Attachment 4: 24-04-2021_540am_SpecLayId6.png
24-04-2021_540am_SpecLayId6.png
Attachment 5: 24-04-2021_540am_SpecLayId7.png
24-04-2021_540am_SpecLayId7.png
Attachment 6: 24-04-2021_540am_SpecLayId8.png
24-04-2021_540am_SpecLayId8.png
Attachment 7: 24-04-2021_540am_SpecLayId1.png
24-04-2021_540am_SpecLayId1.png
Attachment 8: 24-04-2021_540am_SpecLayId2.png
24-04-2021_540am_SpecLayId2.png
  273   Sat Apr 24 04:07:23 2021 MLucesb scalers

ucesb is back - see attachment

Attachment 1: 24-04-2021_5am_ucesb.png
24-04-2021_5am_ucesb.png
  272   Sat Apr 24 03:32:11 2021 MLquick checks

Quick system checks after crash

Temperature ok.

Still cannot connect remotely to /ucesb with my gsi web login and password (?).

 

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida05 fault      0x31a2 :      0x31a2 :      0  
aida05 : WR status 0x10
 White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      9      3      4      2      3      2      1      4      2      3      7   : 39804
aida02 :     29      4      2      3      0      3      1      4      2      3      7   : 39828
aida03 :     20      6      1      2      2      3      1      4      2      3      7   : 39888
aida04 :     19      5      4      2      1      3      1      4      2      3      7   : 39860
aida05 :     27      5      4      1      2      3      2      3      1      4      7   : 40692
aida06 :     17     10      3      2      1      3      1      4      2      3      7   : 39876
aida07 :     22      7      2      2      2      3      1      4      2      3      7   : 39920
aida08 :     12      6      5      1      3      2      1      4      2      3      7   : 39824
aida09 :      0      9      1      3      1      3      1      4      2      3      7   : 39800
aida10 :      2      2      2      3      3      2      3      2      2      3      7   : 39256
aida11 :     21      7      4      3      0      3      2      3      2      3      7   : 39596
aida12 :     18      8      3      0      2      3      2      3      2      3      7   : 39608

 

Attachment 1: 24-04-2021_430am_GoodEvent.png
24-04-2021_430am_GoodEvent.png
  271   Sat Apr 24 03:16:19 2021 TDaida-gsi window manager/login session crashed
Looks like the aida-gsi window manager/login session crashed.

On site S460 shift restarted AnyDesk. Upon re-connection *all* browsers, terminals, applications were gone
I have re-started the basics but BNC PB-5 pulser, CAEN NIM bin, MBS rate monitor, system console monitor etc
still need to be re-started. 

Extract from /var/log/messages

Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal invoked oom-killer: gfp_mask=0xd0, order=0, oom_adj=0, oom_score_adj=0
Apr 24 03:19:01 aidas-gsi kernel: gnome-terminal cpuset=/ mems_allowed=0
Apr 24 03:19:01 aidas-gsi kernel: Pid: 18955, comm: gnome-terminal Not tainted 2.6.32-696.el6.x86_64 #1
Apr 24 03:19:01 aidas-gsi kernel: Call Trace:
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131550>] ? dump_header+0x90/0x1b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8123c1dc>] ? security_real_capable_noaudit+0x3c/0x70
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811319d2>] ? oom_kill_process+0x82/0x2a0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131911>] ? select_bad_process+0xe1/0x120
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81131e10>] ? out_of_memory+0x220/0x3c0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113e7ec>] ? __alloc_pages_nodemask+0x93c/0x950
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81015533>] ? native_sched_clock+0x13/0x80
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff810aaea1>] ? lock_hrtimer_base+0x31/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81177a3a>] ? alloc_pages_current+0xaa/0x110
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8113bb4e>] ? __get_free_pages+0xe/0x50
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1944>] ? __pollwait+0xb4/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff81348684>] ? n_tty_poll+0x194/0x1d0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff813440aa>] ? tty_poll+0x8a/0xa0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1db5>] ? do_sys_poll+0x2d5/0x5b0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1890>] ? __pollwait+0x0/0xf0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1980>] ? pollwake+0x0/0x60
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b1745>] ? poll_select_set_timeout+0x95/0xb0
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff811b2281>] ? sys_poll+0x71/0x100
Apr 24 03:19:01 aidas-gsi kernel: [<ffffffff8100b0d2>] ? system_call_fastpath+0x16/0x1b
Apr 24 03:19:01 aidas-gsi kernel: Mem-Info:
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:    0, btch:   1 usd:   0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 172
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  45
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd:  97
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 142
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  49
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd:  56
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 179
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd:  67
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 107
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 170
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  54
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 118
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 171
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 196
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal per-cpu:
Apr 24 03:19:01 aidas-gsi kernel: CPU    0: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    1: hi:  186, btch:  31 usd:  36
Apr 24 03:19:01 aidas-gsi kernel: CPU    2: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU    3: hi:  186, btch:  31 usd: 167
Apr 24 03:19:01 aidas-gsi kernel: CPU    4: hi:  186, btch:  31 usd:  37
Apr 24 03:19:01 aidas-gsi kernel: CPU    5: hi:  186, btch:  31 usd: 128
Apr 24 03:19:01 aidas-gsi kernel: CPU    6: hi:  186, btch:  31 usd: 173
Apr 24 03:19:01 aidas-gsi kernel: CPU    7: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    8: hi:  186, btch:  31 usd: 181
Apr 24 03:19:01 aidas-gsi kernel: CPU    9: hi:  186, btch:  31 usd: 161
Apr 24 03:19:01 aidas-gsi kernel: CPU   10: hi:  186, btch:  31 usd: 177
Apr 24 03:19:01 aidas-gsi kernel: CPU   11: hi:  186, btch:  31 usd: 178
Apr 24 03:19:01 aidas-gsi kernel: CPU   12: hi:  186, btch:  31 usd:  99
Apr 24 03:19:01 aidas-gsi kernel: CPU   13: hi:  186, btch:  31 usd: 183
Apr 24 03:19:01 aidas-gsi kernel: CPU   14: hi:  186, btch:  31 usd: 165
Apr 24 03:19:01 aidas-gsi kernel: CPU   15: hi:  186, btch:  31 usd: 162
Apr 24 03:19:01 aidas-gsi kernel: active_anon:3408671 inactive_anon:401328 isolated_anon:128
Apr 24 03:19:01 aidas-gsi kernel: active_file:22995 inactive_file:61739 isolated_file:0
Apr 24 03:19:01 aidas-gsi kernel: unevictable:0 dirty:59856 writeback:22 unstable:0
Apr 24 03:19:01 aidas-gsi kernel: free:35843 slab_reclaimable:15134 slab_unreclaimable:65491
Apr 24 03:19:01 aidas-gsi kernel: mapped:129529 shmem:249820 pagetables:24586 bounce:0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA free:15752kB min:60kB low:72kB high:88kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB 
slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 874 16024 16024
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32 free:64204kB min:3680kB low:4600kB high:5520kB active_anon:369500kB inactive_anon:377436kB active_file:10988kB 
inactive_file:26828kB unevictable:0kB isolated(anon):384kB isolated(file):0kB present:895084kB mlocked:0kB dirty:26856kB writeback:88kB mapped:35896kB shmem:39156kB 
slab_reclaimable:4224kB slab_unreclaimable:4344kB kernel_stack:3344kB pagetables:2032kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:65792 all_unreclaimable? 
no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 15150 15150
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal free:63416kB min:63836kB low:79792kB high:95752kB active_anon:13265184kB inactive_anon:1227876kB active_file:80992kB 
inactive_file:220128kB unevictable:0kB isolated(anon):128kB isolated(file):0kB present:15513600kB mlocked:0kB dirty:212568kB writeback:0kB mapped:482220kB 
shmem:960124kB slab_reclaimable:56312kB slab_unreclaimable:257620kB kernel_stack:17600kB pagetables:96312kB unstable:0kB bounce:0kB writeback_tmp:0kB 
pages_scanned:17824 all_unreclaimable? no
Apr 24 03:19:01 aidas-gsi kernel: lowmem_reserve[]: 0 0 0 0
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA: 0*4kB 1*8kB 2*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15752kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 DMA32: 373*4kB 559*8kB 428*16kB 296*32kB 189*64kB 111*128kB 43*256kB 5*512kB 2*1024kB 0*2048kB 0*4096kB = 64204kB
Apr 24 03:19:01 aidas-gsi kernel: Node 0 Normal: 16100*4kB 1*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 64408kB
Apr 24 03:19:01 aidas-gsi kernel: 357788 total pagecache pages
Apr 24 03:19:01 aidas-gsi kernel: 22930 pages in swap cache
Apr 24 03:19:01 aidas-gsi kernel: Swap cache stats: add 3720478, delete 3697548, find 5026852/5228898
Apr 24 03:19:01 aidas-gsi kernel: Free swap  = 1826100kB
Apr 24 03:19:01 aidas-gsi kernel: Total swap = 8200188kB
Apr 24 03:19:01 aidas-gsi kernel: 4194303 pages RAM
Apr 24 03:19:01 aidas-gsi kernel: 97868 pages reserved
Apr 24 03:19:01 aidas-gsi kernel: 578727 pages shared
Apr 24 03:19:01 aidas-gsi kernel: 3846484 pages non-shared
Apr 24 03:19:01 aidas-gsi kernel: [ pid ]   uid  tgid total_vm      rss cpu oom_adj oom_score_adj name
Apr 24 03:19:01 aidas-gsi kernel: [ 1388]     0  1388     2819      130   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 3927]     0  3927     7442      128  12     -17         -1000 auditd
Apr 24 03:19:01 aidas-gsi kernel: [ 3961]     0  3961    62914      755  15       0             0 rsyslogd
Apr 24 03:19:01 aidas-gsi kernel: [ 4029]     0  4029     4604      141   7       0             0 irqbalance
Apr 24 03:19:01 aidas-gsi kernel: [ 4054]    32  4054     4746      124  13       0             0 rpcbind
Apr 24 03:19:01 aidas-gsi kernel: [ 4074]    81  4074     8194      440   6       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 4089]     0  4089    26490     2647   4       0             0 NetworkManager
Apr 24 03:19:01 aidas-gsi kernel: [ 4096]     0  4096    14534      328  12       0             0 modem-manager
Apr 24 03:19:01 aidas-gsi kernel: [ 4112]    29  4112     5839       97   0       0             0 rpc.statd
Apr 24 03:19:01 aidas-gsi kernel: [ 4151]     0  4151    47297      117  15       0             0 cupsd
Apr 24 03:19:01 aidas-gsi kernel: [ 4171]     0  4171    11251       75   9       0             0 wpa_supplicant
Apr 24 03:19:01 aidas-gsi kernel: [ 4180]     0  4180     2280      162   9       0             0 dhclient
Apr 24 03:19:01 aidas-gsi kernel: [ 4210]     0  4210     1020       82   8       0             0 acpid
Apr 24 03:19:01 aidas-gsi kernel: [ 4222]    68  4222     9968      569  15       0             0 hald
Apr 24 03:19:01 aidas-gsi kernel: [ 4223]     0  4223     5100      184  12       0             0 hald-runner
Apr 24 03:19:01 aidas-gsi kernel: [ 4271]     0  4271     5630      157  12       0             0 hald-addon-inpu
Apr 24 03:19:01 aidas-gsi kernel: [ 4272]    68  4272     4502      120   0       0             0 hald-addon-acpi
Apr 24 03:19:01 aidas-gsi kernel: [ 4306]     0  4306    96540      161   0       0             0 automount
Apr 24 03:19:01 aidas-gsi kernel: [ 4492]     0  4492    26831       15   9       0             0 rpc.rquotad
Apr 24 03:19:01 aidas-gsi kernel: [ 4497]     0  4497     5450      213   4       0             0 rpc.mountd
Apr 24 03:19:01 aidas-gsi kernel: [ 4542]     0  4542     5776       42   4       0             0 rpc.idmapd
Apr 24 03:19:01 aidas-gsi kernel: [ 4567]     0  4567    16573      113   3     -17         -1000 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 4578]     0  4578     5429      117   7       0             0 xinetd
Apr 24 03:19:01 aidas-gsi kernel: [ 4589]    38  4589     7686      213  10       0             0 ntpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4675]     0  4675    20260      147   5       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [ 4682]    89  4682    20323      141   5       0             0 qmgr
Apr 24 03:19:01 aidas-gsi kernel: [ 4696]     0  4696    27153      253   0       0             0 ksmtuned
Apr 24 03:19:01 aidas-gsi kernel: [ 4709]     0  4709    29222      137  10       0             0 crond
Apr 24 03:19:01 aidas-gsi kernel: [ 4724]     0  4724     5277       50  12       0             0 atd
Apr 24 03:19:01 aidas-gsi kernel: [ 4741]     0  4741   172796     2328   0       0             0 virt-who
Apr 24 03:19:01 aidas-gsi kernel: [ 4762]     0  4762   256599      407  10       0             0 libvirtd
Apr 24 03:19:01 aidas-gsi kernel: [ 4858]     0  4858    16126      150  10       0             0 certmonger
Apr 24 03:19:01 aidas-gsi kernel: [ 4903]    99  4903     3223      102   2       0             0 dnsmasq
Apr 24 03:19:01 aidas-gsi kernel: [ 4918]     0  4918     4539      223   5       0             0 dhcpd
Apr 24 03:19:01 aidas-gsi kernel: [ 4920]     0  4920    33509      312  11       0             0 gdm-binary
Apr 24 03:19:01 aidas-gsi kernel: [ 4927]     0  4927     1016       61  11       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4929]     0  4929     1016       61  12       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4931]     0  4931     1016       61   9       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4933]     0  4933     1016       61  15       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4935]     0  4935     1016       61   1       0             0 mingetty
Apr 24 03:19:01 aidas-gsi kernel: [ 4944]     0  4944     3148      440   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 4945]     0  4945     3148      438   2     -17         -1000 udevd
Apr 24 03:19:01 aidas-gsi kernel: [ 5050]     0  5050  1029198      377   3       0             0 console-kit-dae
Apr 24 03:19:01 aidas-gsi kernel: [ 5126]     0  5126    12508      365   3       0             0 devkit-power-da
Apr 24 03:19:01 aidas-gsi kernel: [ 5164]     0  5164    45435      581   1       0             0 polkitd
Apr 24 03:19:01 aidas-gsi kernel: [ 5175]   499  5175    42114      146   0       0             0 rtkit-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5702]     0  5702    14009      514   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [ 5725]     0  5725    11306       74   1       0             0 udisks-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18391]     0 18391    41240      293   8       0             0 gdm-simple-slav
Apr 24 03:19:01 aidas-gsi kernel: [18393]     0 18393  3476635  2412883  13       0             0 Xorg
Apr 24 03:19:01 aidas-gsi kernel: [18427]    42 18427     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18464]     0 18464    47926      269   0       0             0 gdm-session-wor
Apr 24 03:19:01 aidas-gsi kernel: [18510]   500 18510    57281      424   1       0             0 gnome-keyring-d
Apr 24 03:19:01 aidas-gsi kernel: [18520]   500 18520    64493      751   0       0             0 gnome-session
Apr 24 03:19:01 aidas-gsi kernel: [18528]   500 18528     5017       83   0       0             0 dbus-launch
Apr 24 03:19:01 aidas-gsi kernel: [18529]   500 18529     8211      379   8       0             0 dbus-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18604]   500 18604    33379     1113  13       0             0 gconfd-2
Apr 24 03:19:01 aidas-gsi kernel: [18612]   500 18612   139831     1984   9       0             0 gnome-settings-
Apr 24 03:19:01 aidas-gsi kernel: [18614]   500 18614    71643      696  12       0             0 seahorse-daemon
Apr 24 03:19:01 aidas-gsi kernel: [18616]   500 18616    34351      298  13       0             0 gvfsd
Apr 24 03:19:01 aidas-gsi kernel: [18636]   500 18636   112427     2321   4       0             0 metacity
Apr 24 03:19:01 aidas-gsi kernel: [18637]   500 18637   203008     6568  10       0             0 gnome-panel
Apr 24 03:19:01 aidas-gsi kernel: [18638]   500 18638   326834    16727  10       0             0 nautilus
Apr 24 03:19:01 aidas-gsi kernel: [18640]   500 18640   141364      341  14       0             0 bonobo-activati
Apr 24 03:19:01 aidas-gsi kernel: [18642]   500 18642    65558      741  10       0             0 bluetooth-apple
Apr 24 03:19:01 aidas-gsi kernel: [18645]   500 18645    68687      968   4       0             0 gnome-power-man
Apr 24 03:19:01 aidas-gsi kernel: [18646]   500 18646     7601      812  12       0             0 restorecond
Apr 24 03:19:01 aidas-gsi kernel: [18648]   500 18648    28879      401   5       0             0 im-settings-dae
Apr 24 03:19:01 aidas-gsi kernel: [18649]   500 18649   132278     1323   4       0             0 gnome-volume-co
Apr 24 03:19:01 aidas-gsi kernel: [18654]   500 18654    75318     2282  10       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18657]   500 18657   119660     1244   2       0             0 trashapplet
Apr 24 03:19:01 aidas-gsi kernel: [18659]   500 18659    88087     3906   5       0             0 wnck-applet
Apr 24 03:19:01 aidas-gsi kernel: [18660]   500 18660    62988      653   7       0             0 vino-server
Apr 24 03:19:01 aidas-gsi kernel: [18661]   500 18661    72490      958   0       0             0 polkit-gnome-au
Apr 24 03:19:01 aidas-gsi kernel: [18662]   500 18662    82733     1441   5       0             0 nm-applet
Apr 24 03:19:01 aidas-gsi kernel: [18666]   500 18666   126400      635   1       0             0 gvfs-gdu-volume
Apr 24 03:19:01 aidas-gsi kernel: [18667]   500 18667    74660     1042   5       0             0 gpk-update-icon
Apr 24 03:19:01 aidas-gsi kernel: [18669]   500 18669    64033      850   0       0             0 gdu-notificatio
Apr 24 03:19:01 aidas-gsi kernel: [18676]   500 18676    81107     2344   6       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [18693]   500 18693   131702     2471   3       0             0 pulseaudio
Apr 24 03:19:01 aidas-gsi kernel: [18823]   500 18823    58170      269   4       0             0 gvfs-afc-volume
Apr 24 03:19:01 aidas-gsi kernel: [18836]   500 18836    37789      300   8       0             0 gvfs-gphoto2-vo
Apr 24 03:19:01 aidas-gsi kernel: [18844]   500 18844    36051      437   0       0             0 gvfsd-trash
Apr 24 03:19:01 aidas-gsi kernel: [18868]   500 18868    10275      287   6       0             0 gconf-im-settin
Apr 24 03:19:01 aidas-gsi kernel: [18869]   500 18869    84055     4872   3       0             0 gnome-screensav
Apr 24 03:19:01 aidas-gsi kernel: [18870]   500 18870     2057       88   3       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18871]   500 18871    27123      198   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18894]   500 18894    98681     1288   0       0             0 gdm-user-switch
Apr 24 03:19:01 aidas-gsi kernel: [18896]   500 18896    72800      911   3       0             0 notification-ar
Apr 24 03:19:01 aidas-gsi kernel: [18897]   500 18897   143660     1846   2       0             0 clock-applet
Apr 24 03:19:01 aidas-gsi kernel: [18900]   500 18900    23718      242   3       0             0 gconf-helper
Apr 24 03:19:01 aidas-gsi kernel: [18918]   500 18918    34358      265   0       0             0 gvfsd-burn
Apr 24 03:19:01 aidas-gsi kernel: [18932]   500 18932    33897      307   4       0             0 gvfsd-metadata
Apr 24 03:19:01 aidas-gsi kernel: [18955]   500 18955    82196     7037   7       0             0 gnome-terminal
Apr 24 03:19:01 aidas-gsi kernel: [18957]   500 18957     2057       97   7       0             0 gnome-pty-helpe
Apr 24 03:19:01 aidas-gsi kernel: [18977]   500 18977    27123      114   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [18991]     0 18991    47361      130  10       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [18992]     0 18992    27524      290  15       0             0 iptraf
Apr 24 03:19:01 aidas-gsi kernel: [21973]   500 21973  1058234   192783   0       0             0 firefox
Apr 24 03:19:01 aidas-gsi kernel: [21987]   500 21987    68608      284   3       0             0 at-spi-bus-laun
Apr 24 03:19:01 aidas-gsi kernel: [22221]   500 22221   731047    34381   0       0             0 Privileged Cont
Apr 24 03:19:01 aidas-gsi kernel: [22248]   500 22248   705574    10970   0       0             0 WebExtensions
Apr 24 03:19:01 aidas-gsi kernel: [22335]   500 22335   960920   166074   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22407]   500 22407  1023120    77418   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [22810]   500 22810    27123      114   2       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [22826]   500 22826    15487      185   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [22830]   500 22830    27123      114   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26234]   500 26234   839272    43450   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [26311]   500 26311    27123      282   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [26884]   500 26884    27123      115   5       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19388]   500 19388    27123      293   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11683]   500 11683   791783    18526   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [ 4876]   500  4876    54985      304   5       0             0 gvfsd-computer
Apr 24 03:19:01 aidas-gsi kernel: [17248]   500 17248    29595      100   1       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17249]   500 17249    29631      157  10       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [17250]   500 17250    27124      114   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17266]   500 17266    15489      181   8       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [19879]     0 19879    25531      154   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19883]   500 19883    25531      163   2       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [19884]   500 19884    27638      106  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [19907]   500 19907    29595       97   5       0             0 screen
Apr 24 03:19:01 aidas-gsi kernel: [14850]   500 14850    56156      327   2       0             0 gvfsd-network
Apr 24 03:19:01 aidas-gsi kernel: [14855]   500 14855    36479      318   3       0             0 gvfsd-dnssd
Apr 24 03:19:01 aidas-gsi kernel: [22615]   500 22615    27123      117   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [13867]     0 13867    25531      161   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13880]   500 13880    25645      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [13881]   500 13881    14400      138   8       0             0 sftp-server
Apr 24 03:19:01 aidas-gsi kernel: [20258]   500 20258   776888    93602   3       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [25910]   500 25910    15050      209   3       0             0 ssh
Apr 24 03:19:01 aidas-gsi kernel: [26010]   500 26010     2879       95  15       0             0 tinyproxy
Apr 24 03:19:01 aidas-gsi kernel: [26513]     0 26513    25530      204  12       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [26517]   500 26517    25530      177   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29203]     0 29203    25016      164  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [29207]   500 29207    25534      158   6       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 1105]   500  1105    27123      143   7       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [17163]     0 17163    25016      190  15       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [17167]   500 17167    25534      185   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18690]     0 18690    25530      183   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18695]   500 18695    25530      138   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [18696]   500 18696    27123      127   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [20802]     0 20802    25530      173   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20806]   500 20806    25530      142   8       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [20807]   500 20807    27123      172   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 9697]     0  9697    25530      187   0       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9701]   500  9701    25563      184  10       0             0 sshd
Apr 24 03:19:01 aidas-gsi kernel: [ 9703]   500  9703    27123      132   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [21737]   500 21737    29350      360  10       0             0 python
Apr 24 03:19:01 aidas-gsi kernel: [21860]   500 21860   166837     1623   5       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21861]   500 21861   208277     3415   0       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [21868]   500 21868   470786     1884   2       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [20297]   500 20297   731145    44730   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [23452]     0 23452    47361      305   5       0             0 sudo
Apr 24 03:19:01 aidas-gsi kernel: [23453]     0 23453    25240       86   2       0             0 tail
Apr 24 03:19:01 aidas-gsi kernel: [23841]   500 23841    27123      243   3       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [  345]   500   345    27156      311  15       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 4825]   500  4825   682055    25908   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [16950]   500 16950    26921      257   1       0             0 watch
Apr 24 03:19:01 aidas-gsi kernel: [10822]   500 10822  1060110   432573   3       0             0 anydesk
Apr 24 03:19:01 aidas-gsi kernel: [29109]   500 29109    27186      482   0       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [ 3754]   500  3754    27123      375   8       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [11083]   500 11083    28030      216   3       0             0 HTTPD@8115
Apr 24 03:19:01 aidas-gsi kernel: [11087]   500 11087    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11089]   500 11089    36731     3211  12       0             0 tclsh8.5_copy2
Apr 24 03:19:01 aidas-gsi kernel: [11097]   500 11097    28030      207   1       0             0 TapeServer
Apr 24 03:19:01 aidas-gsi kernel: [11104]   500 11104     5398     1567  10       0             0 master
Apr 24 03:19:01 aidas-gsi kernel: [11105]   500 11105    25231      134  12       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11106]   500 11106     4323      187   6       0             0 stats
Apr 24 03:19:01 aidas-gsi kernel: [11107]   500 11107     4918      219   5       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11108]   500 11108     4917      194   9       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11109]   500 11109     4917      194  12       0             0 driver
Apr 24 03:19:01 aidas-gsi kernel: [11110]   500 11110     4630      214   0       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11111]   500 11111     4630      182   9       0             0 linkTCP
Apr 24 03:19:01 aidas-gsi kernel: [11114]   500 11114    28030      217  12       0             0 NewMerger
Apr 24 03:19:01 aidas-gsi kernel: [11124]   500 11124   404629     8857   0       0             0 master64
Apr 24 03:19:01 aidas-gsi kernel: [11125]   500 11125    25231      135   9       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: [11126]   500 11126    19761    17318  13       0             0 statrate64
Apr 24 03:19:01 aidas-gsi kernel: [11127]   500 11127    35968    11059   2       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11128]   500 11128    35968    11498  11       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11129]   500 11129    35968    11581  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11130]   500 11130    35968    12366   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11131]   500 11131    35968    10031   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11132]   500 11132    35968    10689   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11133]   500 11133    35968    11896   6       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11134]   500 11134    35968    10510  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11135]   500 11135    35968    15679  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11136]   500 11136    35968    14623  14       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11137]   500 11137    35968    12790  15       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11138]   500 11138    35968    11683   0       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11139]   500 11139    35968     8773   5       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11140]   500 11140    35968     8774  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11141]   500 11141    35968     8772  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11142]   500 11142    35968     8773  10       0             0 link64
Apr 24 03:19:01 aidas-gsi kernel: [11152]   500 11152   307392    47667   5       0             0 merge64
Apr 24 03:19:01 aidas-gsi kernel: [11200]   500 11200   723346    37902   5       0             0 file:// Content
Apr 24 03:19:01 aidas-gsi kernel: [11338]   500 11338    28030      216   0       0             0 HTTPD
Apr 24 03:19:01 aidas-gsi kernel: [11342]   500 11342    27504      195   0       0             0 TclHttpd-server
Apr 24 03:19:01 aidas-gsi kernel: [11344]   500 11344    46965    14788   3       0             0 tclsh8.5
Apr 24 03:19:01 aidas-gsi kernel: [12835]   500 12835   766232    71066   0       0             0 Web Content
Apr 24 03:19:01 aidas-gsi kernel: [24071]   500 24071    27123      409   9       0             0 bash
Apr 24 03:19:01 aidas-gsi kernel: [24347]   500 24347    28030      231   4       0             0 datarelaymbs
Apr 24 03:19:01 aidas-gsi kernel: [24351]   500 24351     7751     1313  10       0             0 DataRelayMBS
Apr 24 03:19:01 aidas-gsi kernel: [  407]    89   407    20280      470  13       0             0 pickup
Apr 24 03:19:01 aidas-gsi kernel: [ 1586]     0  1586     4238      156  13       0             0 anacron
Apr 24 03:19:01 aidas-gsi kernel: [ 1876]     0  1876    25231      144  10       0             0 sleep
Apr 24 03:19:01 aidas-gsi kernel: Out of memory: Kill process 18393 (Xorg) score 523 or sacrifice child
Apr 24 03:19:01 aidas-gsi kernel: Killed process 18393, UID 0, (Xorg) total-vm:13906540kB, anon-rss:9558800kB, file-rss:92712kB
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:17 aidas-gsi gnome-keyring-daemon[18510]: dbus failure unregistering from session: Connection is closed
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11471" (uid=500 pid=1884 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:30 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11473" (uid=500 pid=1887 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:45 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:19:47 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1961 of process 1961 (/usr/bin/pulseaudio) owned by '42' high priority at nice level -11.
Apr 24 03:19:48 aidas-gsi pulseaudio[1961]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1965 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1966 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:48 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 1967 of process 1961 (/usr/bin/pulseaudio) owned by '42' RT at priority 5.
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11485" (uid=500 pid=1981 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:19:52 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11487" (uid=500 pid=1983 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: DNS-SD initialization failed: Daemon not running
Apr 24 03:30:44 aidas-gsi seahorse-daemon[2215]: init gpgme version 1.1.8
Apr 24 03:30:44 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2243 of process 2243 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:45 aidas-gsi pulseaudio[2243]: conf-parser.c: [/usr/share/pulseaudio/alsa-mixer/paths/hdmi-output-0.conf:6] Unknown lvalue 'required-any' in section 'Jack 
HDMI/DP,pcm=3'.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2301 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2302 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:45 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2303 of process 2243 (/usr/bin/pulseaudio) owned by '500' RT at priority 5.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2325 of process 2325 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2325]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2326 of process 2326 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2326]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11506" (uid=500 pid=2328 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:46 aidas-gsi rtkit-daemon[5175]: Sucessfully made thread 2418 of process 2418 (/usr/bin/pulseaudio) owned by '500' high priority at nice level -11.
Apr 24 03:30:46 aidas-gsi pulseaudio[2418]: pid.c: Daemon already running.
Apr 24 03:30:46 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11508" (uid=500 pid=2420 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:30:51 aidas-gsi pulseaudio[2243]: ratelimit.c: 25 events suppressed
Apr 24 03:30:51 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.251 from b8:27:eb:bb:46:7b via p4p1
Apr 24 03:32:31 aidas-gsi dhcpd: DHCPACK on 192.168.11.251 to b8:27:eb:bb:46:7b via p4p1
Apr 24 03:33:11 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:38:36 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11521" (uid=500 pid=3005 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11524" (uid=500 pid=3012 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11525" (uid=500 pid=3015 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11527" (uid=500 pid=3018 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11529" (uid=500 pid=3022 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:39:51 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11531" (uid=500 pid=3024 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:13 aidas-gsi dbus: [system] Rejected send message, 2 matched rules; type="method_call", sender=":1.11533" (uid=500 pid=3100 comm="/usr/libexec/anydesk) 
interface="org.freedesktop.DBus.Properties" member="GetAll" error name="(unset)" requested_reply=0 destination=":1.20" (uid=0 pid=5050 comm="/usr/sbin/console-kit-
daemon))
Apr 24 03:40:37 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:40:46 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:41:22 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:42:19 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:43:31 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:47:10 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.117 from b8:27:eb:40:53:e8 via p4p1
Apr 24 03:49:55 aidas-gsi dhcpd: DHCPACK on 192.168.11.117 to b8:27:eb:40:53:e8 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.7 (192.168.11.99) from d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.7 to d8:80:39:41:b4:0c via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.4 (192.168.11.99) from d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.4 to d8:80:39:41:f6:b7 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.3 (192.168.11.99) from d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.3 to d8:80:39:41:d8:21 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.8 (192.168.11.99) from d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.8 to d8:80:39:41:ba:2b via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.2 (192.168.11.99) from d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.2 to d8:80:39:41:ba:22 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.1 (192.168.11.99) from d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.1 to d8:80:39:41:ba:8a via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.6 (192.168.11.99) from d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.6 to d8:80:39:41:ee:72 via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.5 (192.168.11.99) from d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi dhcpd: DHCPACK on 192.168.11.5 to d8:80:39:41:d7:cc via p4p1
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:816 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida08 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:952 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida07 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:884 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:895 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida02 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:892 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida01 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:839 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:50:59 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:897 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida05 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.8
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.1
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.7
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.3
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.4
Apr 24 03:51:03 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.2
Apr 24 03:51:04 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.5
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:29 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:51:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:06 aidas-gsi kernel: nouveau: evo channel stalled
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.9 (192.168.11.99) from d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.9 to d8:80:39:41:f6:ee via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.10 (192.168.11.99) from d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:10 aidas-gsi dhcpd: DHCPACK on 192.168.11.10 to d8:80:39:41:ba:89 via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.11 (192.168.11.99) from d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.11 to d8:80:39:41:f6:5a via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPDISCOVER from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPOFFER on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPREQUEST for 192.168.11.12 (192.168.11.99) from d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi dhcpd: DHCPACK on 192.168.11.12 to d8:80:39:41:cf:ac via p4p1
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:814 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida09 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:852 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida10 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:694 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida11 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:770 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida12 
(/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.9
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.10
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.12
Apr 24 03:52:15 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.11
Apr 24 03:52:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 24 03:52:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 
(/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
  270   Sat Apr 24 03:13:05 2021 MLAIDA DAQ Reset

AIDA PC crashe completely at ~3:30am CET.

Tom was called and he Restarted the AIDA system at ~4am with Run 52.

Not sure what cause the crash but AnyDesk remote connection was suddenly lost and once reconnected all Terminal sessions were gone.

 

  269   Sat Apr 24 01:44:09 2021 MLsystem wide checks

system wide checks:

 

 

Clock status:
FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White Rabbit

         Base         Current     Difference
aida06 fault      0x679f :      0x67a4 :      5  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error page not working !!

Memory information:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     15      4      6      4      3      2      4      2      3      6   : 36420
aida02 :     36     25     24      6      0      4      2      3      2      3      4   : 28056
aida03 :     17      7      4      3      3      5      2      2      3      3      6   : 36444
aida04 :      4      5      4      5      3      4      2      3      3      3      6   : 36824
aida05 :     23      9      4      2      3      3      1      3      3      3      6   : 36452
aida06 :     26      8      5      1      3      4      2      2      3      3      6   : 36312
aida07 :     21      9      3      3      4      5      1      2      3      3      6   : 36268
aida08 :     27      9      2      2      2      4      1      3      3      3      6   : 36500
aida09 :     13      4      3      3      1      2      4      2      3      3      6   : 36388
aida10 :      1     10      0      3      1      2      3      3      2      3      6   : 35572
aida11 :     23     13      8      3      3      4      3      3      2      3      6   : 36196
aida12 :     19      7      4      3      2      3      2      3      3      3      6   : 36644

 

Attachment 1: 24-04-2021_230am_AIDA_HV.png
24-04-2021_230am_AIDA_HV.png
Attachment 2: 24-04-2021_230am_ucesbStat.png
24-04-2021_230am_ucesbStat.png
Attachment 3: 24-04-2021_230am_grafana.png
24-04-2021_230am_grafana.png
Attachment 4: 24-04-2021_230am_AidaGoodEvent.png
24-04-2021_230am_AidaGoodEvent.png
Attachment 5: 24-04-2021_230am_Correlation#8.png
24-04-2021_230am_Correlation#8.png
Attachment 6: 24-04-2021_230am_RESUME#3.png
24-04-2021_230am_RESUME#3.png
Attachment 7: 24-04-2021_230am_PAUSE#2.png
24-04-2021_230am_PAUSE#2.png
  268   Fri Apr 23 20:19:52 2021 TDFriday 23 April 20.00-00.00
21.19 system wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      7      5      7      3      3      2      4      2      3      6   : 36356
aida02 :     67     29     27      4      0      4      2      3      2      3      4   : 28196
aida03 :     14      5      2      4      3      5      2      2      3      3      6   : 36416
aida04 :      5      5      0      7      2      3      3      3      3      3      6   : 36892
aida05 :     26      5      4      3      3      3      1      3      3      3      6   : 36464
aida06 :     20      7      5      2      3      4      2      2      3      3      6   : 36312
aida07 :     24      7      1      2      3      4      2      3      3      3      6   : 36776
aida08 :     22      7      6      2      3      3      1      3      3      3      6   : 36464
aida09 :     20      7      1      2      1      3      3      2      3      3      6   : 36248
aida10 :      9      1      6      1      1      2      2      4      2      3      6   : 35820
aida11 :     15      4      2      3      3      4      3      3      2      3      6   : 35996
aida12 :     22     11      6      2      2      3      2      2      3      3      6   : 36176


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Apr 23 03:51:08 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

21.20 DSSSD biases & leakage currents OK - attachment 1
      FEE64 tempos OK - attachment 2
      good event stats - attachment 3
      Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4

23.35 System wide checks - no changes to above
      DSSSD biases & leakage currents OK - attachment 5
      FEE64 tempos OK - attachment 6
      good event stats - attachment 7

23.46 DAQ continues file S460/R51_378
Attachment 1: 201.png
201.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(7).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(7).png
Attachment 3: Screenshot_2021-04-23_Statistics_aidas-gsi(7).png
Screenshot_2021-04-23_Statistics_aidas-gsi(7).png
Attachment 4: Screenshot_2021-04-23_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-23_AIDA_Alerting_-_Grafana.png
Attachment 5: 202.png
202.png
Attachment 6: Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
Screenshot_2021-04-23_Statistics_aidas-gsi(8).png
  267   Fri Apr 23 15:13:28 2021 DSJ23 April 16.00-20.00 shift

16:00 taking over from Muneerah

16:15 - temp, ucesb, database, stats etc. look ok.

16:41 - Nic restarting MBS

17:05 - temp, ucesb, database, stats etc. look ok.

17:43 - temp, ucesb, database, stats etc. look ok.

 

 

18.10 - system wide checks.  All same as previous checks - eg elog 264 + 266

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

    
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

 

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
 

Check FPGA Timestamp Error - HTML errors

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19      7      4      7      4      3      2      4      2      3      6   : 36388
aida02 :      1      2      5      2      0      4      3      4      3      4      3   : 27044
aida03 :     28      4      4      4      4      4      2      2      3      3      6   : 36432
aida04 :      5      5      2      6      2      3      3      3      3      3      6   : 36892
aida05 :     24      4      7      3      3      3      2      2      3      3      6   : 36240
aida06 :     21      6      6      3      3      4      2      2      3      3      6   : 36356
aida07 :     15      7      4      3      3      4      2      2      3      3      6   : 36308
aida08 :     23      6      7      1      3      4      1      3      3      3      6   : 36572
aida09 :      1      8      2      1      1      2      4      2      3      3      6   : 36292
aida10 :      3      4      6      2      2      2      3      3      2      3      6   : 35660
aida11 :     15      7      4      1      2      3      2      4      2      3      6   : 36052
aida12 :     25     13      5      2      2      3      2      3      3      3      6   : 36700
 

18:45 - temp, ucesb, database, stats etc. look ok.

19:03 - temp, ucesb, database, stats etc. look ok.

19:26  - temp, ucesb, database, stats etc. look ok.

TapeData (df -h) - 7.2T  3.9T  2.9T  58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7

 

19:30 - beam being shared with R3B, implantation rates fluctuation by~  50% back to normal 19:40

 

19:50 system checks - all as previous

 

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
 

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

    
         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Check FPGA Timestamp Error - HTML errors

 

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     25      6      4      7      4      3      2      4      2      3      6   : 36404
aida02 :     20      7      5      1      0      4      2      3      2      3      4   : 27384
aida03 :     28      6      3      5      3      5      2      2      3      3      6   : 36528
aida04 :     19      8      4      6      2      4      4      2      3      3      6   : 36876
aida05 :      7      3      8      3      4      3      1      3      3      3      6   : 36500
aida06 :     25      8      4      3      3      4      2      3      3      3      6   : 36868
aida07 :     18      7      3      4      3      4      2      2      3      3      6   : 36336
aida08 :     16     10      6      2      3      3      1      3      3      3      6   : 36464
aida09 :      0      3      1      1      1      3      4      2      3      3      6   : 36360
aida10 :      8      1     10      0      2      2      2      4      2      3      6   : 35912
aida11 :      2      2      2      3      3      4      3      3      2      3      6   : 35928
aida12 :     12      7      4      1      3      3      2      3      3      3      6   : 36616
 

Attachment 1: Screenshot_2021-04-23_Statistics_aidas-gsi(5).png
Screenshot_2021-04-23_Statistics_aidas-gsi(5).png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(5).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(5).png
Attachment 3: 555.png
555.png
Attachment 4: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(6).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(6).png
Attachment 5: Screenshot_2021-04-23_Statistics_aidas-gsi(6).png
Screenshot_2021-04-23_Statistics_aidas-gsi(6).png
Attachment 6: 556.png
556.png
  266   Fri Apr 23 13:13:46 2021 MA, TDFriday 23 April 12.00-16.00
14.15 System wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp errors - reports 'server internal error'

14.16 DSSSD bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachment 2
      good events stats - attachment 3

14.22 DAQ continues file S460/R51_205
      beam intensity 2.2e+09/s


15:53 system check
Rates, Temperatures, Voltages are ok and attached 4,5,6


Clock check

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC check

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



White Rabbit


		 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR




FPGA
Loge page error!

Memory check
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      6      2      1      2      3      2      4      2      3      6   : 36044
aida02 :     15      3      2      2      0      3      2      3      2      3      4   : 27188
aida03 :     29      4      0      3      2      4      2      2      3      3      6   : 36212
aida04 :      4      6      4      4      2      3      3      3      3      3      6   : 36864
aida05 :     19      5      4      2      3      3      1      3      3      3      6   : 36404
aida06 :     24      9      5      2      3      3      3      2      3      3      6   : 36472
aida07 :     20      4      9      0      2      3      2      2      3      3      6   : 36096
aida08 :     22      9      4      3      1      3      1      3      3      3      6   : 36352
aida09 :     20      7      3      2      2      3      3      2      3      3      6   : 36344
aida10 :     25      9      9      5      2      3      2      3      3      3      6   : 36828
aida11 :     24     11      2      0      1      3      1      3      3      3      6   : 36248
aida12 :     26      5      4      2      1      3      2      2      3      3      6   : 36048
Attachment 1: 200.png
200.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 3: Screenshot_2021-04-23_Statistics_aidas-gsi(4).png
Screenshot_2021-04-23_Statistics_aidas-gsi(4).png
Attachment 4: 210423_1550_Voltages.png
210423_1550_Voltages.png
Attachment 5: 210423_1550_Stats.png
210423_1550_Stats.png
Attachment 6: 210423_1550_Temp.png
210423_1550_Temp.png
  265   Fri Apr 23 12:12:08 2021 TDaida02 restart - 03.13UTC+2 Friday 23 April
Overnight aida02 crashed and restarted itself. A power cycle of all FEE64s was required to recover.

The system log appears to show the aida02 embedded processor crashed (note log times are UTC+1
for some reason).

Apr 21 00:41:21 aida02 kernel: xaida: open:
Apr 23 02:09:41 aida02 kernel: Unable to handle kernel paging request for data at address 0x00200200
Apr 23 02:09:41 aida02 kernel: Faulting instruction address: 0xc0091dd8
Apr 23 02:09:41 aida02 kernel: Oops: Kernel access of bad area, sig: 11 [#1]
Apr 23 02:09:41 aida02 kernel: PREEMPT Xilinx Virtex440
Apr 23 02:09:41 aida02 kernel: Modules linked in: aidamem xdriver xh_spidev_register
Apr 23 02:09:42 aida02 kernel: NIP: c0091dd8 LR: c0091e40 CTR: 00000000
Apr 23 02:09:42 aida02 kernel: REGS: c6835e60 TRAP: 0300   Not tainted  (2.6.31)
Apr 23 02:09:42 aida02 kernel: MSR: 00021000 <ME,CE>  CR: 24000088  XER: 00000000
Apr 23 02:09:42 aida02 kernel: DEAR: 00200200, ESR: 00800000
Apr 23 02:09:42 aida02 kernel: TASK = c6824880[5] 'events/0' THREAD: c6834000
Apr 23 02:09:42 aida02 kernel: GPR00: c0091e40 c6835f10 c6824880 00000001 00000003 c038c3a8 00000000 c0498000
Apr 23 02:09:42 aida02 kernel: GPR08: 00000008 00100100 00000051 00200200 00485e6c 00005aa8 4805de90 00000001
Apr 23 02:09:42 aida02 kernel: GPR16: 00000000 00000000 4801d000 00000003 c031aef4 c02fb040 c02fb008 00000002
Apr 23 02:09:42 aida02 kernel: GPR24: c6834000 00100100 00200200 c680ae00 00000001 c680daf0 c680dae0 c6940020
Apr 23 02:09:42 aida02 kernel: NIP [c0091dd8] drain_freelist+0x54/0x110
Apr 23 02:09:42 aida02 kernel: LR [c0091e40] drain_freelist+0xbc/0x110
Apr 23 02:09:42 aida02 kernel: Call Trace:
Apr 23 02:09:42 aida02 kernel: [c6835f10] [c0091e40] drain_freelist+0xbc/0x110 (unreliable)
Apr 23 02:09:42 aida02 kernel: [c6835f40] [c0093158] cache_reap+0x104/0x138
Apr 23 02:09:42 aida02 kernel: [c6835f60] [c0048908] worker_thread+0x140/0x204
Apr 23 02:09:42 aida02 kernel: [c6835fc0] [c004cfc8] kthread+0x78/0x7c
Apr 23 02:09:42 aida02 kernel: [c6835ff0] [c000e140] kernel_thread+0x4c/0x68
Apr 23 02:09:42 aida02 kernel: Instruction dump:
Apr 23 02:09:43 aida02 kernel: 3ba40010 7f80e800 419e00d4 3f200010 3f400020 54380024 63390100 635a0200
Apr 23 02:09:43 aida02 kernel: 3b800000 48000068 813f0000 817f0004 <912b0000> 91690004 933f0000 935f0004
Apr 23 02:09:43 aida02 kernel: ---[ end trace 6baa5b82e4ef4e92 ]---
Apr 23 02:09:43 aida02 kernel: note: events/0[5] exited with preempt_count 1
Apr 23 02:13:39 aida02 syslogd 1.4.2: restart.
Apr 23 02:13:39 aida02 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Apr 23 02:13:39 aida02 kernel: Using Xilinx Virtex440 machine description
Apr 23 02:13:39 aida02 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
  264   Fri Apr 23 07:09:57 2021 LJWFriday 23rd April 08:00-12:00

08:30

System Checks

 

Clock Status error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration error:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status error:


         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

 

Statistics - See attacnment 1

Temperatures - See attachment 2

Bias & Leakage Currents - See attachment 3

 

10:02

System Checks:

Clock Status Error:

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration Error:

   
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 4

Temperatures - See attachment 5

Bias & Leakage Currents - See attachment 6

 

10:45

System Checks

Clock Status Error:


FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


ADC Calibration check Error:


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Decoder Status Error:

         Base         Current     Difference
aida06 fault      0x679f :      0x67a2 :      3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error:

Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'

Statistics - See attacnment 7

Temperatures - See attachment 8

Bias & Leakage Currents - See attachment 9

 

Attachment 1: 2304211_1.png
2304211_1.png
Attachment 2: 230421_2.png
230421_2.png
Attachment 3: 230421_3.png
230421_3.png
Attachment 4: 230421_4.png
230421_4.png
Attachment 5: 230421_5.png
230421_5.png
Attachment 6: 230421_6.png
230421_6.png
Attachment 7: 230421_7.png
230421_7.png
Attachment 8: 230421_8.png
230421_8.png
Attachment 9: 230421_9.png
230421_9.png
  263   Thu Apr 22 23:17:46 2021 DJ Friday 23rd April 00:00-08:00

Took over from TD ... All is well.

00:45 System Wide Checks

Clock: 12 passed, 0 failed.

ADC Calibration: Passed 12, 0 failed.

White Rabbit Decoder: Passed 4,failed 8. - See screenshot.

FPGA Passed 11, failed 1. - See screenshot.

FEE64 Linux: See screenshot.

-----

02:49 run closed. Checking beam.

03:00 File 55 opened.

03:13:23 DAQ crashed due to AIDA02 rebooting. TD restarted DAQ. Ready at approx 04:00.

File 56 opened at 4:24.

---------

04:30 System Wide Checks (See screenshots)

White Rabbit Decoder: Passed 12, failed 0.

FPGA: page not loading properly.

----

05:37 System wide checks. See screenshots

Clock: Status: 11 passed. 1 fail.

ADC calibration: Passed 9, failed 3.

White Rabbit decoder: Passed 12, failed 0.

-----

07:32 System wide checks (See screenshots).

 

 

Attachment 1: Screenshot_2021-04-23_Statistics_aidas-gsi.png
Screenshot_2021-04-23_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot-CAEN_HV_Supply.png
Screenshot-CAEN_HV_Supply.png
Attachment 4: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi.png
Attachment 5: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(1).png
Attachment 6: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(2).png
Attachment 7: Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Screenshot_2021-04-23_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 60.png
60.png
Attachment 10: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(4).png
Attachment 12: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(5).png
Attachment 13: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-23_Spectrum_Browser_aidas-gsi.png
Attachment 15: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 16: Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Screenshot_2021-04-23_Statistics_aidas-gsi(2).png
Attachment 17: 61.png
61.png
Attachment 18: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(6).png
Attachment 19: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(7).png
Attachment 20: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(8).png
Attachment 21: Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-23_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 22: Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Screenshot_2021-04-23_Statistics_aidas-gsi(3).png
Attachment 23: 63.png
63.png
Attachment 24: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(9).png
Attachment 25: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(10).png
Attachment 26: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(11).png
Attachment 27: Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
Screenshot_2021-04-23_System_wide_Checks_aidas-gsi(12).png
  262   Thu Apr 22 15:18:20 2021 TDThursday 22 April 16.00-00.00
16.20 zero all histograms, statistics & merger statistics

      most recent merger error messages

MERGE Data Link (29984): bad timestamp  6 3 0xc1ac7f1c 0x0dce7672 0x00002cda5dce7672 0x16782cda5dce7672 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc18b7e62 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1957ebc 0x0dce7e42 0x00002cda5dce7e42 0x16782cda5dce7e42 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1b67ecb 0x0dcf0ae2 0x00002cda5dcf0ae2 0x16782cda5dcf0ae2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1957ea3 0x0dcf12b2 0x00002cda5dcf12b2 0x16782cda5dcf12b2 0x16782cda5dcf2032
MERGE Data Link (29984): bad timestamp  6 3 0xc1867fba 0x0b2ee142 0x00002dc9eb2ee142 0x16782dc9eb2ee142 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp  6 3 0x86800889 0x0ad28158 0x00002dc9eb2ee142 0x16782dc9ead28158 0x16782dc9fad22532
MERGE Data Link (29984): bad timestamp  6 3 0xc1be7e6e 0x0b2ef0e2 0x00002dc9eb2ef0e2 0x16782dc9eb2ef0e2 0x16782dc9fad2c172
MERGE Data Link (29984): bad timestamp  6 3 0xc1b08006 0x0b2f5e42 0x00002dc9eb2f5e42 0x16782dc9eb2f5e42 0x16782dc9fad34502
MERGE Data Link (29984): bad timestamp  6 3 0xc1807de7 0x0ad3c342 0x00002dc9eb2f6de2 0x16782dc9ead3c342 0x16782dc9fad34502
MERGE Statistics counters now cleared

16.21 merger server terminal reset & clear

      TapeData -> /media/ThirdDRive/TapeData

[npg@aidas-gsi bin64]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_aidasgsi-lv_root
                       50G   13G   34G  28% /
tmpfs                 7.9G  559M  7.3G   7% /dev/shm
/dev/sda2             477M   42M  410M  10% /boot
/dev/sda1             200M  264K  200M   1% /boot/efi
/dev/mapper/vg_aidasgsi-lv_home
                      401G   79G  302G  21% /home
/dev/mapper/vg_aidas2-lv_home
                      7.2T  3.9T  2.9T  58% /media/1e121361-83d3-4825-b6ae-8700b07e0ca7
/dev/sdd1             7.2T  6.3T  605G  92% /media/SecondDrive
/dev/sdc1             7.2T  3.6T  3.3T  52% /media/ThirdDrive

      at current rate (10.5Mb/s) we have 3.6d remaining 
      
17.40 *all* system wide checks OK *except* FPGA timestamp errors

		 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x51 : 	 81  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Apr 21 01:51:54 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

17.43 FEE64 temps OK - attachment 1
      statistics - attachments 2-8
      per FEE64 rate spectra - attachments 9 & 10
      1.8.L spectra - attachments 11-14
      1.8.H spectra - attachments 15-16
      1.8.W spectra 20us FSR - attachments 17-18
      1.8.W spectra 200us, 2ms & 20ms FSR - attachments 19-21, 30-32
      merger, tape server & merger statistics 19-25
      ucesb - attachment 26
      Grafana DSSD biases & leakage currents - attachment 27
      taperserver & tapeserver/merger server terminals - no timewarp errors since c. 15.25 - attachment 28
      DSSSD biases & leakage currents - attachment 29


22.17 *all* system wide checks OK *except*

		 Base 		Current 	Difference
aida01 fault 	 0x3612 : 	 0x3613 : 	 1  
aida02 fault 	 0x523c : 	 0x523d : 	 1  
aida03 fault 	 0xc5cd : 	 0xc5ce : 	 1  
aida04 fault 	 0x14f1 : 	 0x14f2 : 	 1  
aida05 fault 	 0x9a4e : 	 0x9a4f : 	 1  
aida06 fault 	 0xfdd2 : 	 0xfdd3 : 	 1  
aida07 fault 	 0x4a0 : 	 0x4a1 : 	 1  
aida08 fault 	 0xc796 : 	 0xc797 : 	 1  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x57 : 	 87  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Wed Apr 21 01:51:54 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

Most recent merger errors

MERGE Data Link (29979): bad timestamp  1 3 0xc05080ef 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc06f80d7 0x0c88aaa2 0x000047aa2c88aaa2 0x167847aa2c88aaa2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc0778222 0x0c88b272 0x000047aa2c88b272 0x167847aa2c88b272 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc0598098 0x0c88e152 0x000047aa2c88e152 0x167847aa2c88e152 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc042815b 0x0c894eb2 0x000047aa2c894eb2 0x167847aa2c894eb2 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc05980eb 0x0c899502 0x000047aa2c899502 0x167847aa2c899502 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc075813c 0x0c89e322 0x000047aa2c89e322 0x167847aa2c89e322 0x167847aa2c8a34d2
MERGE Data Link (29979): bad timestamp  1 3 0xc04280d5 0x0c89f2c2 0x000047aa2c89f2c2 0x167847aa2c89f2c2 0x167847aa2c8a34d2
MERGE Actor (29995): Working with 0 from 12 data sources
MERGE Actor (29995): Working with 0 from 12 data sources

22.20 DSSSD biases & leakage currents - attachment 33
      FEE64 temps OK - attachment 34
      statistics - attachments 35
      merger statistics 36

      
Attachment 1: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 2: Screenshot_2021-04-22_Statistics_aidas-gsi(4).png
Screenshot_2021-04-22_Statistics_aidas-gsi(4).png
Attachment 3: Screenshot_2021-04-22_Statistics_aidas-gsi(5).png
Screenshot_2021-04-22_Statistics_aidas-gsi(5).png
Attachment 4: Screenshot_2021-04-22_Statistics_aidas-gsi(6).png
Screenshot_2021-04-22_Statistics_aidas-gsi(6).png
Attachment 5: Screenshot_2021-04-22_Statistics_aidas-gsi(7).png
Screenshot_2021-04-22_Statistics_aidas-gsi(7).png
Attachment 6: Screenshot_2021-04-22_Statistics_aidas-gsi(8).png
Screenshot_2021-04-22_Statistics_aidas-gsi(8).png
Attachment 7: Screenshot_2021-04-22_Statistics_aidas-gsi(9).png
Screenshot_2021-04-22_Statistics_aidas-gsi(9).png
Attachment 8: Screenshot_2021-04-22_Statistics_aidas-gsi(10).png
Screenshot_2021-04-22_Statistics_aidas-gsi(10).png
Attachment 9: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi.png
Attachment 10: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(1).png
Attachment 11: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(2).png
Attachment 12: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(3).png
Attachment 13: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(4).png
Attachment 14: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(5).png
Attachment 15: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(6).png
Attachment 16: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(7).png
Attachment 17: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(8).png
Attachment 18: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(9).png
Attachment 19: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(10).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(10).png
Attachment 20: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(11).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(11).png
Attachment 21: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(12).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(12).png
Attachment 22: Screenshot_2021-04-22_NewMerger_Control_aidas-gsi.png
Screenshot_2021-04-22_NewMerger_Control_aidas-gsi.png
Attachment 23: Screenshot_2021-04-22_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2021-04-22_Tape_Service_(Expert)_aidas-gsi.png
Attachment 24: Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi.png
Attachment 25: Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi(1).png
Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi(1).png
Attachment 26: Screenshot_2021-04-22_ucesb.png
Screenshot_2021-04-22_ucesb.png
Attachment 27: Screenshot_2021-04-22_AIDA_Alerting_-_Grafana(1).png
Screenshot_2021-04-22_AIDA_Alerting_-_Grafana(1).png
Attachment 28: 101.png
101.png
Attachment 29: 100.png
100.png
Attachment 30: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi.png
Attachment 31: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(1).png
Attachment 32: Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-22_Spectrum_Browser_aidas-gsi(2).png
Attachment 33: 102.png
102.png
Attachment 34: Screenshot_2021-04-22_Statistics_aidas-gsi.png
Screenshot_2021-04-22_Statistics_aidas-gsi.png
Attachment 35: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi.png
Attachment 36: Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi.png
Screenshot_2021-04-22_NewMerger_Statistics_aidas-gsi.png
  261   Thu Apr 22 11:03:57 2021 JSThursday 22nd April: 12:00-16:00
12:00 Taking over from Liam, all is well.

12:10 Beam dropped out for a minute, implantation rate is down a bit, peaking around 300 Hz

12:19 Beam stopped (from GSI log "Run stopped because of a call from HKR stating they have a problematic device which they 
need to check and will need apprx. 15 min.")

12:30
ucesb ok (no beam)
DB No faults
Stats ok.

12:34 In order to fix some ongoing FRS problem no beam expected for at least an hour.

13:00
We are taking intermittent beam while they work on the FRS

DB No faults
ucesb ok  elog:261/1
Current ok elog:261/2
Stats ok  elog:261/3
Temps ok elog:261/4

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x46 : 	 70  
FPGA Timestamp error counter test result: Passed 11, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      6      5      3      2      2      2      3      2      3      7   : 39580
aida02 :     21      7      4      2      2      3      2      3      2      3      7   : 39692
aida03 :     20     10      3      3      0      3      2      3      2      3      7   : 39600
aida04 :      8      7      4      5      2      4      3      2      2      3      7   : 39608
aida05 :     17     14      1      2      3      3      4      2      2      3      7   : 39748
aida06 :     17     10      2      3      2      2      4      2      2      3      7   : 39572
aida07 :     27     10      8      7      2      3      3      2      2      3      7   : 39708
aida08 :      9      6      2      3      2      1      3      3      2      3      7   : 39636
aida09 :      1      2      1      4      0      3      3      2      2      3      7   : 39204
aida10 :      5      8      4      4      2      2      1      3      2      3      7   : 39316
aida11 :      6      4      5      3      2      3      2      2      2      3      7   : 39144
aida12 :     15      5      8      1      0      4      1      3      2      3      7   : 39428


13:32
ucesb ok
DB No faults
Stats ok

14:01
ucesb ok
DB No faults
Stats ok

14.30 analysis of file /TapeData/S460/R50_528 - attachment 5
      cat /var/log/messages | grep /MIDAS - attachments 6 & 7
      Grafana DSSSD bias & leakage currents for last 7 days - attachment 8

14:55
DB No faults 
ucesb ok
Current ok elog:261/9
Stats ok  elog:261/10
Temps ok elog:261/11

FPGA Timestamp Check Error:
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x4d : 	 77  
FPGA Timestamp error counter test result: Passed 11, Failed 1


15:30
ucesb ok
DB No faults
Stats ok

15:59
ucesb ok elog:261/12
DB No faults
Stats ok
Attachment 1: 51.png
51.png
Attachment 2: 52.png
52.png
Attachment 3: Screenshot_2021-04-22_Statistics_aidas-gsi(2).png
Screenshot_2021-04-22_Statistics_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 5: R50_528
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256073749 ( 1415213.2 Hz)
        Other data format:    5846251 (   32309.8 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32674 (     180.6 Hz)
                              WR48-63:      32674 (     180.6 Hz)
                           FEE64 disc:    1081787 (    5978.6 Hz)
                             MBS info:    4699116 (   25970.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     342915 (    1895.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      180.944 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               53.074
  2                0.000               66.167
  3                0.000               47.044
  4                0.000              162.945
  5                0.000              122.260
  6                0.000              104.653
  7                0.000              108.241
  8                0.000               75.998
  9                0.000                7.078
 10                0.000               27.522
 11                0.000               39.393
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   18754736     101009          0          0          0          0       2455       2455      95016       1083          0      35321
  1   22221860      61355          0          0          0          0       2741       2741      55873          0          0      20378
  2   16643754    1970924          0          0          0          0       2367       2367     197444    1768746          0      33880
  3   21495935    1857330          0          0          0          0       2886       2886      82623    1768935          0      33490
  4    3525133     403992          0          0          0          0        456        456      99315     303765          0      35967
  5   10189380     355501          0          0          0          0       1262       1262      56604     296373          0      20229
  6   12085720     387518          0          0          0          0       1592       1592     108226     276108          0      33721
  7   11376753     370078          0          0          0          0       1497       1497      82978     284106          0      32056
  8   43639711      66166          0          0          0          0       5406       5406      55354          0          0      18854
  9   42033424      67528          0          0          0          0       5337       5337      56854          0          0      18886
 10   29147293     115191          0          0          0          0       3615       3615     107961          0          0      31625
 11   24960050      89659          0          0          0          0       3060       3060      83539          0          0      28508
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:52321.719s (    0.612 blocks/s,   0.038 Mb/s)
Attachment 6: log2
Apr 13 17:43:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:38 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:40 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:43:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:44:56 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:44:56 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:44:58 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 13 17:45:01 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 16 00:58:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:38 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:38 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:10:43 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:11:08 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:11:09 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:11:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:11:12 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 17 06:33:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Attachment 7: log1
Apr 19 08:33:03 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:36 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:37 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:38 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:40 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:46 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:13:56 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:14:48 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:14:48 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:14:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:14:51 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:51 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:52 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:53 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:29:53 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:30:00 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:31:16 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:31:17 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:31:17 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:31:17 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 09:33:24 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:24 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:24 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:26 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:27 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:27:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:28:43 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:28:45 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:28:45 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 19 10:28:46 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:09 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:09 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:11 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:01:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:02:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:02:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:02:41 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:02:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:04:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:52 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:55 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 00:05:57 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:10:52 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:01 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:05 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:06 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:10 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:13 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:19 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:19 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:18:26 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:21:20 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 08:21:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:28 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:34 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:39 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:41 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:47:44 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 10:50:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:08:07 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:41 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:41 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:42 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:46 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:30:55 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 11:33:56 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:07:57 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:17 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:24 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:25 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:34:31 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:35:30 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:35:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:35:33 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 20 22:35:35 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:20 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.5:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:20 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.3:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:21 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.4:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:22 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.8:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:23 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.1:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:28 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:47 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.7:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:49 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.9:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.12:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:50 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.10:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:36:52 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.11:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Apr 21 01:40:07 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.2:857 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Attachment 8: Screenshot_2021-04-22_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-22_AIDA_Alerting_-_Grafana.png
Attachment 9: 53.png
53.png
Attachment 10: Screenshot_2021-04-22_Statistics_aidas-gsi(3).png
Screenshot_2021-04-22_Statistics_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 12: 54.png
54.png
  260   Thu Apr 22 07:06:37 2021 LPGThursday 22nd April: 08:00-12:00
08:01 - Taking over from Corrigan...

ucesb ok!
DB ok!
Stats ok!

08:30

ucesb ok!
DB ok!
Stats ok!

09:15

ucesb ok! elog:260/4
DB ok!
Stats ok!

Leakage currents ok! elog:260/1
Temperatures ok! elog:260/2
MIDAS Rates ok! elog:260/3

System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x40 : 64
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!


09:45

ucesb ok!
DB ok!
Stats ok!

10:30

ucesb ok!
DB ok!
Stats ok!


11:00

ucesb ok! elog:260/5
DB ok!
Stats ok!

Leakage currents ok! elog:260/6
Temperatures ok! elog:260/7
MIDAS Rates ok! elog:260/8

System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x42 : 66
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!


11:30

ucesb ok! elog:260/5
DB ok!
Stats ok!

11:55

ucesb ok!
DB ok!
Stats ok!
Attachment 1: lpg20.png
lpg20.png
Attachment 2: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-22_Statistics_aidas-gsi.png
Screenshot_2021-04-22_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-22_at_08.19.51.png
Screenshot_2021-04-22_at_08.19.51.png
Attachment 5: Screenshot_2021-04-22_at_09.57.10.png
Screenshot_2021-04-22_at_09.57.10.png
Attachment 6: lpg21.png
lpg21.png
Attachment 7: Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-22_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-22_Statistics_aidas-gsi(1).png
Screenshot_2021-04-22_Statistics_aidas-gsi(1).png
  259   Wed Apr 21 23:05:39 2021 CA22nd April 00:00 - 08:00
00:07 stats/ucesb/DB checks all ok

00:30 all system wide checks ok *except* fpga errors

      			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1f : 	 31  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     statistics ok - attachment 1

     temperatures ok - attachment 2

     detector bias / leakage currents ok - attachment 3

     no error messages in DB terminal & ucesb ok - attachment 4

01:00 stats/ucesb/DB checks all ok

01:30 stats/ucesb/DB checks all ok

01:53 issues with beam dropping in and out - ESR working to stabilise this

02:00 stats/ucesb/DB checks all ok

02:09 beam back in a stable state

02:30 all system wide checks ok *except* fpga errors

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x29 : 	 41  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:33 statistics ok - attachment 5

      temperatures ok - attachment 6

      detector bias / leakage currents ok - attachment 7

      no error messages in DB terminal & ucesb ok - attachment 8

02:45 beam off - potentially serious issue, awaiting update from FRS

03:00 significant issues with UNILAC, beam not expected back for a while

03:05 stats/ucesb/DB checks all ok

03:32 stats/ucesb/DB checks all ok

03:55 beam is back

04:00 stats/ucesb/DB checks all ok

04:30 all system wide checks ok *except* fpga errors

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x2b : 	 43  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

04:32 statistics ok - attachment 9

      temperatures ok - attachment 10

      detector bias / leakage currents ok - attachment 11

      no error messages in DB terminal & ucesb ok - attachment 12

05:00 stats/ucesb/DB checks all ok

05:30 stats/ucesb/DB checks all ok

06:00 stats/ucesb/DB checks all ok

06:30 all system wide checks ok *except* fpga errors

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x34 : 	 52  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

06:32 temperatures ok - attachment 13

      statistics ok - attachment 14

      detector bias / leakage currents ok - attachment 15

      no error messages in DB terminal & ucesb ok - attachment 16

07:00 stats/ucesb/DB checks all ok

07:30 stats/ucesb/DB checks all ok

07:59 stats/ucesb/DB checks all ok



      
Attachment 1: Screenshot_from_2021-04-21_23-31-22.png
Screenshot_from_2021-04-21_23-31-22.png
Attachment 2: Screenshot_from_2021-04-21_23-31-44.png
Screenshot_from_2021-04-21_23-31-44.png
Attachment 3: Screenshot_from_2021-04-21_23-32-03.png
Screenshot_from_2021-04-21_23-32-03.png
Attachment 4: Screenshot_from_2021-04-21_23-34-46.png
Screenshot_from_2021-04-21_23-34-46.png
Attachment 5: Screenshot_from_2021-04-22_01-31-14.png
Screenshot_from_2021-04-22_01-31-14.png
Attachment 6: Screenshot_from_2021-04-22_01-31-37.png
Screenshot_from_2021-04-22_01-31-37.png
Attachment 7: Screenshot_from_2021-04-22_01-34-03.png
Screenshot_from_2021-04-22_01-34-03.png
Attachment 8: Screenshot_from_2021-04-22_01-34-31.png
Screenshot_from_2021-04-22_01-34-31.png
Attachment 9: Screenshot_from_2021-04-22_03-32-36.png
Screenshot_from_2021-04-22_03-32-36.png
Attachment 10: Screenshot_from_2021-04-22_03-32-57.png
Screenshot_from_2021-04-22_03-32-57.png
Attachment 11: Screenshot_from_2021-04-22_03-33-11.png
Screenshot_from_2021-04-22_03-33-11.png
Attachment 12: Screenshot_from_2021-04-22_03-33-57.png
Screenshot_from_2021-04-22_03-33-57.png
Attachment 13: Screenshot_from_2021-04-22_05-33-01.png
Screenshot_from_2021-04-22_05-33-01.png
Attachment 14: Screenshot_from_2021-04-22_05-33-19.png
Screenshot_from_2021-04-22_05-33-19.png
Attachment 15: Screenshot_from_2021-04-22_05-33-30.png
Screenshot_from_2021-04-22_05-33-30.png
Attachment 16: Screenshot_from_2021-04-22_05-33-43.png
Screenshot_from_2021-04-22_05-33-43.png
  258   Wed Apr 21 15:01:14 2021 LJW, MAWednesday 21st April 16:00-20:00

16:30

All System checks okay, except for:

FPGA Timestamp error:

   
             Base         Current         Difference
aida12 fault      0x0 :      0x1b :      27  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Temperature Checks: See attachment 1

Statistics Checks: See attachment 2

Bias & Leakage Currents: See attachment 3

 

18:09

All system checks okay, except for:

FPGA Timestamp error:

   
             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Temperature checks: See attachment 4

Statistics Checks: See attachment 5

Bias & Leakage Currents: See attachment 6

 

19:40

All system checks okay, except for:

FPGA Timestamp error:

             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Temperature checks: See attachment 7

Statistics Checks: See attachment 8

Bias & Leakage Currents: See attachment 9

 

*Beam being tuned to another experiment (R3B) - all DSSD implants fluctuating back and forth to 0 Hz for around the last hour or so*

22:30

General check

Rates, Voltages, Temperatures, Ucesb are attached 10,11,12,13

Its been noticed that aida04 temp. is incresing to about 65 (red), still below 70.

the system wide check are all ok except FPGA


             Base         Current         Difference
aida12 fault      0x0 :      0x1d :      29  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Attachment 1: 210421_1.png
210421_1.png
Attachment 2: 210421_2.png
210421_2.png
Attachment 3: 2104211_3.png
2104211_3.png
Attachment 4: 210421_4.png
210421_4.png
Attachment 5: 210421_5.png
210421_5.png
Attachment 6: 210421_6.png
210421_6.png
Attachment 7: 210421_7.png
210421_7.png
Attachment 8: 210421_8.png
210421_8.png
Attachment 9: 210421_9.png
210421_9.png
Attachment 10: 210421_2230_Stats.png
210421_2230_Stats.png
Attachment 11: 210421_2230_Temp.png
210421_2230_Temp.png
Attachment 12: 210421_2230_Ucesb.png
210421_2230_Ucesb.png
Attachment 13: 210421_2230_Voltages.png
210421_2230_Voltages.png
  257   Wed Apr 21 11:27:55 2021 OHWed 21st April 12:00-16:00
12:30 MBS has crashed
      This time AIDA is still running fine.

      Stats ok - attachment 1
      Temperature - attachment 2
      Bias and leakage currents - attachment 3
     
      All system wide checks ok *except*
	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x19 : 	 25  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Now also note errors on aida7 and aida03 in the merger TS errors
     (0 represents the sum across all FEE)
0	1355800	0x0014b018	
1	0	0x00000000	
2	0	0x00000000	
3	33	0x00000021
4	0	0x00000000	
5	0	0x00000000	
6	0	0x00000000	
7	10	0x0000000a
8	0	0x00000000	
9	0	0x00000000	
10	0	0x00000000	
11	1355757	0x0014afed

15:30 CA takes over

      Stats ok - attachment 4
      Temperature - attachment 5
      Bias and leakage currents - attachment 6

      all system wide checks ok *except*

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x19 : 	 27  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

15:42 db check and ucesb ok
Attachment 1: 210421_1338_Stats.png
210421_1338_Stats.png
Attachment 2: 210421_1328_Temp.png
210421_1328_Temp.png
Attachment 3: 210421_1342_Bias.png
210421_1342_Bias.png
Attachment 4: Screenshot_from_2021-04-21_14-33-36.png
Screenshot_from_2021-04-21_14-33-36.png
Attachment 5: Screenshot_from_2021-04-21_14-34-29.png
Screenshot_from_2021-04-21_14-34-29.png
Attachment 6: Screenshot_from_2021-04-21_14-36-08.png
Screenshot_from_2021-04-21_14-36-08.png
  256   Wed Apr 21 06:56:42 2021 MSWednesday 21 April 08:00-12:00 2021

8:00


 

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
 

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

             Base         Current         Difference
aida12 fault      0x0 :      0x15 :      21  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19      6      7      2      2      4      3      2      2      3      7   : 39596
aida02 :     25      3      2      1      1      3      2      3      2      3      7   : 39548
aida03 :     28      5      3      1      2      2      2      3      2      3      7   : 39528
aida04 :     27      6      2      2      2      4      2      4      2      3      7   : 40316
aida05 :      9      9      4      3      1      3      2      3      3      3      7   : 40652
aida06 :     21      7      6      1      2      2      4      2      2      3      7   : 39564
aida07 :     20      7      5      0      1      2      2      3      2      3      7   : 39448
aida08 :     21      5      4      1      1      2      3      3      2      3      7   : 39708
aida09 :      1      3      1      3      0      2      1      4      2      3      7   : 39564
aida10 :     20      6      3      2      1      2      3      3      2      3      7   : 39728
aida11 :     18      4      2      1      2      2      2      2      2      3      7   : 38952
aida12 :     19      8      7      1      1      2      3      3      2      3      7   : 39772

 

From the merger statistics during the night there have been 1355757 TS errors in aida12. Corrigan said they have been quiet since around 3am though.



10:00

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

    
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 1, Failed 11

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

    
             Base         Current         Difference
aida12 fault      0x0 :      0x15 :      21  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     15      7      6      2      1      4      3      2      2      3      7   : 39508
aida02 :      1      2      1      0      1      2      2      4      2      3      7   : 39780
aida03 :     23      8      2      0      1      3      2      3      2      3      7   : 39548
aida04 :     23      9      1      2      3      3      2      4      2      3      7   : 40244
aida05 :     14      7      2      2      4      3      2      3      1      4      7   : 40784
aida06 :     18      5      5      2      2      2      3      3      2      3      7   : 39808
aida07 :     27      4      3      2      2      3      3      2      2      3      7   : 39420
aida08 :     23      6      3      2      2      2      3      3      2      3      7   : 39804
aida09 :     18      4      6      3      2      4      3      2      2      3      7   : 39592
aida10 :      6      9      5      3      2      3      2      3      2      3      7   : 39696
aida11 :     16      4      1      2      3      4      1      3      2      3      7   : 39536
aida12 :     18      5      7      1      0      3      3      3      2      3      7   : 39808

 

10:22

NH Noticed that the ASIC clocks had not been synchronised following the last AIDA restart. - attachment 7
As the beam is having issues they are not currently recording
I have now synchronised the ASIC clocks.
All ADC are now calibrated

 

11:06

Used the beam off time to change the drive writing to. Now writing to /media/ThirdDrive/TapeData/S460/R50

11:10

Beam is back they will start writing to file again


12:00

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

    
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

    
             Base         Current         Difference
aida12 fault      0x0 :      0x17 :      23  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     20      5      5      1      1      4      2      3      2      3      7   : 39720
aida02 :     16      8      4      1      0      2      1      4      2      3      7   : 39648
aida03 :      9      7      3      2      3      3      2      3      2      3      7   : 39692
aida04 :      8      7      5      2      3      5      3      2      2      3      7   : 39720
aida05 :     20      5      3      4      2      3      3      3      2      3      7   : 39976
aida06 :     25      8      0      3      1      2      3      3      2      3      7   : 39748
aida07 :     24      8      4      2      3      3      2      3      2      3      7   : 39776
aida08 :     22      9      0      2      2      2      3      3      2      3      7   : 39776
aida09 :      1      4      4      3      2      3      2      3      2      3      7   : 39620
aida10 :     13      4      4      3      2      3      1      3      2      3      7   : 39412
aida11 :      2      3      1      1      3      4      2      2      2      3      7   : 39184
aida12 :     16     10      7      2      1      3      1      3      2      3      7   : 39424

 

Attachment 1: Screenshot_2021-04-21_Statistics_aidas-gsi.png
Screenshot_2021-04-21_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 46.png
46.png
Attachment 4: Screenshot_2021-04-21_Statistics_aidas-gsi(1).png
Screenshot_2021-04-21_Statistics_aidas-gsi(1).png
Attachment 5: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 6: 47.png
47.png
Attachment 7: Capture.PNG
Capture.PNG
Attachment 8: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 9: Screenshot_2021-04-21_Statistics_aidas-gsi(2).png
Screenshot_2021-04-21_Statistics_aidas-gsi(2).png
Attachment 10: 48.png
48.png
  255   Tue Apr 20 23:08:22 2021 CAWed Apr 21 00:00-08:00
00:00 - CA takes over

00:15 - stats/database/ucesb ok - attachment 1

      - another burst of bad timestamp error messages in NewMerger terminal - but all FEE64 still ok - attachment 2

      - Merger/TapeService ok

00:21 - beam still off while UNILAC having problems

00:26 - beam is back - rate spectra - attachment 3

00:32 - ucesb following beam return - attachment 4

      - implant rates reaching highs of ~1kHz

00:35 - Grafana leakage current monitor last 6 hours - attachment 5

01:00 *all* system wide checks ok

      stats ok - attachment 6

      temperatures ok - attachment 7

      detector bias / leakage currents ok - attachment 8

      db check ok / ucesb ok

      still observe periodic bursts of bad timestamp errors in NewMerger terminal - FEE64s still linked with merger ok

01:30 AIDA crashes just as I make check, most FEE64 lost connection -> powercycle and restart MIDAS

01:53 AIDA reset completed - writing to file NULL/R48

01:56 all system wide checks ok *except* 

FPGA error counter

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


02:01 good event statistics ok - attachment 9

      temperatures ok - attachment 10

      detector bias / leakage currents ok - attachment 11

      db check / ucesb ok

02:20 beam stop

02:21 beam back

02:30 stats/ucesb/db checks all ok

03:00 stats/ucesb/db checks all ok - attachment 12

03:30 stats/ucesb/db checks all ok

04:00 system wide checks ok, except;

      ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

     FPGA errors

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x9 : 	 9  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

04:03 beam off briefly

04:05 good event statistics ok - attachment 13

04:07 beam back

      temperatures ok - attachment 14

      bias / leakage currents ok - attachment 15

      db check / ucesb ok - attachment 16

      implant rates in AIDA peaking at ~1.5kHz

04:30 stats/ucesb/db checks all ok

04:48 no beam

04:58 beam back

05:00 stats/ucesb/db checks all ok - attachment 17

05:30 stats/ucesb/db checks all ok

06:00 system wide checks ok, except;

      ADC calibration

FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

     FPGA errors

Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x9 : 	 9  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

    FEE64 Linux Memory Info

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      5      3      3      2      3      2      3      2      3      7   : 39628
aida02 :     22     11      3      0      2      3      2      3      2      3      7   : 39648
aida03 :     20     10      1      1      2      2      1      4      2      3      7   : 39760
aida04 :     17     10      2      2      3      3      2      3      2      3      7   : 39732
aida05 :      1      8      3      2      3      2      2      3      3      3      7   : 40564
aida06 :      5      6      2      3      2      2      3      3      2      3      7   : 39748
aida07 :     18      4      7      0      1      3      1      4      2      3      7   : 39832
aida08 :     22     10      3      3      2      2      3      3      2      3      7   : 39864
aida09 :      3      6      2      2      0      3      1      4      2      3      7   : 39708
aida10 :      2      4      3      2      1      2      2      3      2      3      7   : 39384
aida11 :     20      8      3      1      1      3      3      2      2      3      7   : 39328
aida12 :     16      6      3      1      0      3      2      3      2      3      7   : 39488

06:02 good event statistics ok - attachment 18

      FEE64 temperatures ok - attachment 19

      detector bias / leakage currents ok - attachment 20

06:06 db check and ucesb ok - attachment 21

      NewMerger terminal hasn't flagged up any more errors since ~ 03:00

06:30 stats/ucesb/db checks all ok

      DESPEC restarts timesorter

07:00 stats/ucesb/db checks all ok

07:30 stats/ucesb/db checks all ok - attachment 22

      grafana leakage currents last 1 hour - attachment 23

   

    







      
Attachment 1: Screenshot_from_2021-04-20_23-15-20.png
Screenshot_from_2021-04-20_23-15-20.png
Attachment 2: Screenshot_from_2021-04-20_23-18-14.png
Screenshot_from_2021-04-20_23-18-14.png
Attachment 3: Screenshot_from_2021-04-20_23-29-06.png
Screenshot_from_2021-04-20_23-29-06.png
Attachment 4: Screenshot_from_2021-04-20_23-32-08.png
Screenshot_from_2021-04-20_23-32-08.png
Attachment 5: Screenshot_from_2021-04-20_23-34-42.png
Screenshot_from_2021-04-20_23-34-42.png
Attachment 6: Screenshot_from_2021-04-21_00-00-06.png
Screenshot_from_2021-04-21_00-00-06.png
Attachment 7: Screenshot_from_2021-04-21_00-00-29.png
Screenshot_from_2021-04-21_00-00-29.png
Attachment 8: Screenshot_from_2021-04-21_00-01-15.png
Screenshot_from_2021-04-21_00-01-15.png
Attachment 9: Screenshot_from_2021-04-21_01-00-03.png
Screenshot_from_2021-04-21_01-00-03.png
Attachment 10: Screenshot_from_2021-04-21_01-00-48.png
Screenshot_from_2021-04-21_01-00-48.png
Attachment 11: Screenshot_from_2021-04-21_01-01-05.png
Screenshot_from_2021-04-21_01-01-05.png
Attachment 12: Screenshot_from_2021-04-21_01-59-30.png
Screenshot_from_2021-04-21_01-59-30.png
Attachment 13: Screenshot_from_2021-04-21_03-05-24.png
Screenshot_from_2021-04-21_03-05-24.png
Attachment 14: Screenshot_from_2021-04-21_03-05-47.png
Screenshot_from_2021-04-21_03-05-47.png
Attachment 15: Screenshot_from_2021-04-21_03-06-07.png
Screenshot_from_2021-04-21_03-06-07.png
Attachment 16: Screenshot_from_2021-04-21_03-11-44.png
Screenshot_from_2021-04-21_03-11-44.png
Attachment 17: Screenshot_from_2021-04-21_04-00-22.png
Screenshot_from_2021-04-21_04-00-22.png
Attachment 18: Screenshot_from_2021-04-21_05-00-18.png
Screenshot_from_2021-04-21_05-00-18.png
Attachment 19: Screenshot_from_2021-04-21_05-00-43.png
Screenshot_from_2021-04-21_05-00-43.png
Attachment 20: Screenshot_from_2021-04-21_05-03-47.png
Screenshot_from_2021-04-21_05-03-47.png
Attachment 21: Screenshot_from_2021-04-21_05-05-43.png
Screenshot_from_2021-04-21_05-05-43.png
Attachment 22: Screenshot_from_2021-04-21_06-30-20.png
Screenshot_from_2021-04-21_06-30-20.png
Attachment 23: Screenshot_from_2021-04-21_06-32-51.png
Screenshot_from_2021-04-21_06-32-51.png
  254   Tue Apr 20 19:00:00 2021 JSTuesday 20th 20:00-00:00
20:00  

Taking over from Liam, all systems good.

20:30  

DB No faults.
Stats good
ucesb ok

20:58 

DB No faults.
Stats good
Temps good
ucesb ok

21:31 

DB No faults.
Stats good
Temps good
ucesb ok

22:02

DB No faults.
Stats good elog:254/3
Temps good elog:254/4
ucesb ok elog:254/1
Current ok elog:254/2

ADC Calibration check:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1

22:06 ucesb - DAQ Error - Onsite team checking daq crash
Tape server icon stopped spinning
Stats Rates 0, Counter aida01 zeroed but others normal. elog:254/5

Called Tom. Tried resetting in midas, no luck. Full reboot of DAQ and midas required.
According to log at 22:07 aida01 reset
Data taking stopped at 22:04 file R45_193

22:50
After a full power cycle all appear to be up and running well. Starting full checks.

23:00

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0

Check FPGA is having a page error

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	
4k
aida01 :      4      8      2      2      1      3      1      4      2      3      7   : 
39792
aida02 :     13      9      2      4      2      3      2      3      2      3      7   : 
39708
aida03 :     19      5      4      3      1      3      1      4      2      3      7   : 
39892
aida04 :     14      5      2      2      0      5      2      3      2      3      7   : 
39744
aida05 :     16     11      3      2      2      4      3      2      1      4      7   : 
40584
aida06 :     15      9      5      2      2      2      1      4      2      3      7   : 
39828
aida07 :     15      9      3      3      2      2      1      4      2      3      7   : 
39828
aida08 :     21      4      3      4      2      2      1      4      2      3      7   : 
39844
aida09 :     11      6      1      2      3      4      2      3      2      3      7   : 
39788
aida10 :      0      6      4      2      0      3      2      3      2      3      7   : 
39472
aida11 :     16      7      2      0      1      3      2      3      2      3      7   : 
39512
aida12 :      3      7      4      1      3      2      2      3      2      3      7   : 
39524

DB No faults.
Stats good elog:254/6
Temps good elog:254/7 aida01 had been high during reboot issue, >70, but now is back down
Current ok elog:254/8
ucesb ok

23:25 

DB No faults.
Stats good
Temps good
ucesb ok

23:42 No Beam
23:48 Beam Back

23:57

DB No faults.
Stats good elog:254/11
Temps good elog:254/12
ucesb ok elog:254/9
Current ok elog:254/10

Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
	
*FPGA check page error*

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      9      5      2      1      2      1      4      2      3      7   : 39812
aida02 :     13      9      5      2      2      2      1      4      2      3      7   : 39820
aida03 :     21      8      3      2      2      2      1      4      2      3      7   : 39812
aida04 :     13      4      3      1      1      5      2      3      2      3      7   : 39780
aida05 :     24      5      3      2      2      4      2      3      2      3      7   : 39800
aida06 :     16      5      5      2      2      2      1      4      2      3      7   : 39800
aida07 :     22      6      2      2      1      3      1      4      2      3      7   : 39848
aida08 :     21      8      2      3      2      2      1      4      2      3      7   : 39828
aida09 :     18      7      1      1      3      4      2      3      2      3      7   : 39792
aida10 :     19      3      3      1      0      3      2      3      2      3      7   : 39476
aida11 :     17      4      4      0      1      3      2      3      2      3      7   : 39524
aida12 :     13      4      3      2      3      1      2      3      2      3      7   : 39428

00:04 Handing over to Corrigan
Attachment 1: 40.png
40.png
Attachment 2: 41.png
41.png
Attachment 3: Screenshot_2021-04-20_Statistics_aidas-gsi(3).png
Screenshot_2021-04-20_Statistics_aidas-gsi(3).png
Attachment 4: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-20_Statistics_aidas-gsi(4).png
Screenshot_2021-04-20_Statistics_aidas-gsi(4).png
Attachment 6: Screenshot_2021-04-20_Statistics_aidas-gsi(5).png
Screenshot_2021-04-20_Statistics_aidas-gsi(5).png
Attachment 7: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 8: 42.png
42.png
Attachment 9: 44.png
44.png
Attachment 10: 45.png
45.png
Attachment 11: Screenshot_2021-04-20_Statistics_aidas-gsi(6).png
Screenshot_2021-04-20_Statistics_aidas-gsi(6).png
Attachment 12: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(5).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(5).png
  253   Tue Apr 20 14:56:38 2021 LPGTuesday 20th 16:00-20:00
16:00

New shift, ready to go! Beam stopped just as I started... Waiting for update.
Beam back at 16:08, ucesb scalers shown in elog:253/1

Database ok!
Stats ok!
ucesb ok!

Temperatures ok! elog:253/2
Statistics ok! elog:253/3
Leakage currents ok! Steadily rising, day/night effect? - elog:253/4 and elog:253/5

16:21

Online histograms zeroed in MIDAS. Same done in MBS DAQ.

16:45

Database ok!
Stats ok!
ucesb ok!

System wide checks:
Clock ok!
ADC 1 failed:
FEE64 module aida06 failed
WR ok!
FPGA ok!
Memory ok!

17:15

Database ok!
Stats ok!
ucesb ok!

17:45

Database ok!
Stats ok!
ucesb ok!

18:15

Database ok!
Stats ok!
ucesb ok!

System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!

MIDAS Rates ok! elog:253/6
Temperatures ok! elog:253/7
Leakage currents ok, but still slowly rising! Topped out at ~20:30 yesterday elog:253/8


18:42
Message in the Merger terminal window (elog:253/9)
MERGE Actor (15671): Working with 0 from 12 data sources


19:05

Database ok!
Stats ok!
ucesb ok!

19:30

Some MIDAS spectra in elog:253/10 and elog:253/11

Database ok!
Stats ok!
ucesb ok!

19:50

Database ok!
Stats ok!
ucesb ok! elog:253/12

System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!

MIDAS Rates ok! elog:253/13
Temperatures ok! elog:253/14
Leakage currents ok, now plateaued! elog:253/15


19:55 - Handing over to James Smile
Attachment 1: Screenshot_2021-04-20_at_15.09.52.png
Screenshot_2021-04-20_at_15.09.52.png
Attachment 2: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-20_Statistics_aidas-gsi.png
Screenshot_2021-04-20_Statistics_aidas-gsi.png
Attachment 4: lpg1.png
lpg1.png
Attachment 5: Screenshot_2021-04-20_at_15.13.14.png
Screenshot_2021-04-20_at_15.13.14.png
Attachment 6: Screenshot_2021-04-20_Statistics_aidas-gsi.png
Screenshot_2021-04-20_Statistics_aidas-gsi.png
Attachment 7: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 8: lpg2.png
lpg2.png
Attachment 9: lpg3_merger.png
lpg3_merger.png
Attachment 10: Screenshot_2021-04-20_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-20_Spectrum_Browser_aidas-gsi.png
Attachment 11: Screenshot_2021-04-20_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-20_Spectrum_Browser_aidas-gsi(1).png
Attachment 12: Screenshot_2021-04-20_at_18.50.56.png
Screenshot_2021-04-20_at_18.50.56.png
Attachment 13: Screenshot_2021-04-20_Statistics_aidas-gsi(2).png
Screenshot_2021-04-20_Statistics_aidas-gsi(2).png
Attachment 14: Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-20_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 15: lpg3.png
lpg3.png
  252   Tue Apr 20 10:54:41 2021 CBApril 20 12:00 - 16:00 shift
11:55 CA got AIDA back online

12:00 Stats OK (attach 1)
No faults found
UCESB OK - no beam atm (attach 2)
Temps OK (running hotter than yesterday, but not worrying yet) - attach 3
Bias OK (attach 4)

System wide checks
Clock - *all* pass
ADC - 06 fails
WR - *all* pass
FPGA *all* pass

Memory info
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      4      1      3      2      1      2      4      2      3      7   : 39876
aida02 :     23      6      1      2      2      2      1      4      2      3      7   : 39772
aida03 :     24      4      4      1      2      3      1      4      2      3      7   : 39904
aida04 :     10      3      1      1      1      3      1      4      2      3      7   : 39728
aida05 :     12      2      3      2      1      4      2      3      1      4      7   : 40688
aida06 :     10      5      4      2      2      3      3      4      2      3      7   : 40400
aida07 :     22      6      1      3      1      3      1      4      2      3      7   : 39864
aida08 :     20      4      4      1      2      3      2      4      2      3      7   : 40144
aida09 :      1      2      2      3      1      2      1      3      3      3      7   : 40148
aida10 :      3      3      1      2      2      2      2      3      2      3      7   : 39412
aida11 :     25      7      0      1      0      3      2      3      2      3      7   : 39484
aida12 :     21      7      3      1      3      2      2      3      2      3      7   : 39580


Waiting for beam.

12:42 Stats OK, UCESB ok, No faults. No beam.
13:21 Stats OK, UCESB ok, No faults. No beam.
13:24 Implants in DSSD#3. ASIC check. Disappeared. No beam.

14:00 No beam.
Stats OK (attach 5)
Temps OK (attach 6)
UCESB OK
Bias  OK (attach 7)

System wide checks
Clock OK
ADC calibration 06 fails
WR OK
FPGA OK

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9      4      1      3      2      3      3      2      3      7   : 39848
aida02 :     15      8      2      2      2      3      2      3      2      3      7   : 39644
aida03 :      8      1      2      3      4      3      2      3      2      3      7   : 39720
aida04 :     25     12      3      1      1      3      1      4      2      3      7   : 39892
aida05 :     13      4      4      4      0      3      2      3      1      4      7   : 40596
aida06 :     21      8      3      3      0      3      3      4      2      3      7   : 40356
aida07 :     17     11      0      1      1      3      1      4      2      3      7   : 39804
aida08 :     16      8      2      3      2      3      1      4      2      3      7   : 39936
aida09 :     17      6      3      2      1      2      1      4      2      3      7   : 39716
aida10 :      3      3      2      2      4      1      2      2      3      3      7   : 39940
aida11 :      4      4      4      1      1      3      3      2      2      3      7   : 39248
aida12 :      2      4      1      3      1      2      2      3      2      3      7   : 39384

14:30 Stats OK, UCESB ok, No faults. No beam.
15:00 Stats OK, UCESB ok, No faults. No beam.


15:24 Beam is back!
Stats OK (attach 8), UCESB ok (attach 9), No faults.

15:50 Shift handed over to Liam
Attachment 1: Stats1.png
Stats1.png
Attachment 2: ucesb_nobeam.png
ucesb_nobeam.png
Attachment 3: Temps1.png
Temps1.png
Attachment 4: Bias1.png
Bias1.png
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Temps2.png
Temps2.png
Attachment 7: Bias2.png
Bias2.png
Attachment 8: Stats3.png
Stats3.png
Attachment 9: UCESB2.png
UCESB2.png
  251   Tue Apr 20 07:06:20 2021 CAApril 20th 08:00 - 12:00
08:09 DAQ crashes at start of shift (!)

      again happens at same time as beam being lost

      All FEE64 lost connection

      Beam downtime for next 3-4 hours

      Performed powercycle and reset

08:40 AIDA back up and running, writing to R36

08:42 *all* system wide checks ok

      temperatures ok - attachment 1

      statistics ok - attachment 2

      detector bias/leakage currents ok - attachment 3

08:57 options file size check;


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Tue Apr 20 08:22:14 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

08:59 no error messages in database check terminal
      stats ok

09:16 waveforms - attachments 4 & 5

09:30 no error messages in database check terminal
      stats ok

10:07 no error messages in database check terminal
      stats ok

10:30 aida01, aida02, aida03, aida04, aida07 lost connection, DAQ crashes

      powercycle attempted, but FEE modules remain unconnected

      TD performs further power cycle

11.12 reboot aida05

      DAQ reset/setup

      system wide checks

      sync asic clocks OK
      clock status & fpga timestamp errors OK
      
      WR decoder status

		 Base 		Current 	Difference
aida09 fault 	 0xb307 : 	 0xb307 : 	 0  
aida09 : WR status 0x40
 aida10 fault 	 0x3d6c : 	 0x3d6c : 	 0  
aida10 : WR status 0x40
 aida11 fault 	 0xa610 : 	 0xa610 : 	 0  
aida11 : WR status 0x40
 aida12 fault 	 0x7c7 : 	 0x7c7 : 	 0  
aida12 : WR status 0x40
 White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

NH had access to area, unplugged and replugged HDMI cables from aida09, aida10, aida11, aida12 root to MACB (at both ends)

earlier on, Akash secured HDMI cable to aida07

11:55 AIDA powercycle and reset, DAQ ok now

      writing to file NULL/R45

12:00 CB takes over
Attachment 1: Screenshot_from_2021-04-20_07-45-09.png
Screenshot_from_2021-04-20_07-45-09.png
Attachment 2: Screenshot_from_2021-04-20_07-45-43.png
Screenshot_from_2021-04-20_07-45-43.png
Attachment 3: Screenshot_from_2021-04-20_07-46-05.png
Screenshot_from_2021-04-20_07-46-05.png
Attachment 4: Screenshot_from_2021-04-20_08-14-58.png
Screenshot_from_2021-04-20_08-14-58.png
Attachment 5: Screenshot_from_2021-04-20_08-15-38.png
Screenshot_from_2021-04-20_08-15-38.png
  250   Mon Apr 19 23:01:02 2021 OH20th April 00:00-08:00
23:52 Noticed AIDA07 has dropped out of the merger and is producing no statistics
      Powercycling to recover

00:14 All FEEs recovered from powercycle
      aida09 running again at HEC threshold of 0x2 (Was set to 0x1 earlier in the day while testing the HEC event disparity in DSSD3)
      Now writing to R34
      Following the powercycle the rates in the FEEs are much improved and back to where they were before the 08:30 crash on the 19th. - Attachment 1
      Waveforms also look improved - attachmens 2 and 3


      Current system parameters
      ASIC settings Dec19 (All LEC comparator 0xa)
      Pulser at 2V and 2Hz
      DSSD 1 and 2 Bias 160V
      DSSD 3 at 120V (Will lower to 100 again during the beam off period tomorrow)

     

00:28 In analysis of R34_13 the number of HEC items in DSSD3 appears to make sense again. The number in aida09 are consistent with the number in aida05 and the same for aida11 and aida07.
      This appears to confirm that the earlier HEC issues we were observing were hardware related but we are unsure what/why/how these issues were caused.
      Analysis - attachment 4

00:40 Issue with one of the FRS magnets so they stop recording data. Leaving AIDA running.

01:08 While beam was down have reset DSSD3 bias to 100V as was agreed with Tom and Nic in email
      Leakage current decreased as expected - attachment 5
      Once beam is back will check HEC still as expected.

01:14 Last magnet in FRS has an issue with the water flowing. Either sensor issue or water flow is to low.
      Hakke have increased the water flow to hopefully keep the water running.
      Beam is back.

      From ucesb it seems that the HEC events in DSSD3 are still appearing as expected. Will check a file once we have a complete file.
      Checked R34_36 and the HEC events in DSSD3 are still consistent - attachment 6

2:20 Statistics ok - attachment 7
     Temp - attachment 8
     Bias and leakage current ok - attachment 9

     All system wide checks ok except for WR error check:
	
		 Base 		Current 	Difference
aida07 fault 	 0x9154 : 	 0x915b : 	 7  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 10, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Note aida07 reports two bad timestamps in the merger
     MERGE Data Link (3762): bad timestamp  6 3 0xc1817ea7 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56
     MERGE Data Link (3762): bad timestamp  6 3 0xc1b77e25 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56


02:25 Remembered during a pre-experiment meeting with Vic, Patrick and Carl they mentioned the merger would track timestamp sequence errors
      Can confirm this is true. Can see the two in aida07 in the counter in the merger stats page - attachment 10

02:45 Now up to 7 timestamp errors on aida07 in the merger
      
02:49 Ran R34_45 through AIDASort - It reports a greater number of implant events are able to have a front back match performed
      Ratio between aida07 and 09 is 0.71 -> Double what it was before
      Energy response of the HEC channels in aida07 and aida09 looks equivalent (left group aida09 right group aida11) - attahchment 11 

04:40 Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents - attachment 14
      Clock status, ADC calibration and memory infomration all ok
      aida07 fails both WR and FPGA
	
		 Base 		Current 	Difference
aida07 fault 	 0x915b : 	 0x9161 : 	 6  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 10, Failed 2

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


       TS errors in the merger still only at 7 with all in aida07

06:25 Statistics - attachment 15
      Temps ok - attachment 16
      Bias and leakage currents ok - attachment 17

      System wide checks: clock status, adc calibration and memory information all ok
      WR multiple fails
	
		 Base 		Current 	Difference
aida05 fault 	 0xc9ca : 	 0xc9cb : 	 1  
aida06 fault 	 0xe900 : 	 0xe901 : 	 1  
aida07 fault 	 0x915b : 	 0x9166 : 	 11  
aida08 fault 	 0x641e : 	 0x641f : 	 1  
aida10 fault 	 0x8d3 : 	 0x8d3 : 	 0  
aida10 : WR status 0x10
 White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     FPGA aida07 fails with difference of 1

     No new TS sequence errors in merger      
      

07:10 Beam has dropped out.
07:14 Beam is back
Attachment 1: 210420_0016_Stats.png
210420_0016_Stats.png
Attachment 2: 210420_0021_Layout7.png
210420_0021_Layout7.png
Attachment 3: 210420_0022_Layout8.png
210420_0022_Layout8.png
Attachment 4: 210420_0026_R34_13_Analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255975543 ( 1356743.5 Hz)
        Other data format:    5944457 (   31507.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32703 (     173.3 Hz)
                              WR48-63:      32703 (     173.3 Hz)
                           FEE64 disc:    1073243 (    5688.5 Hz)
                             MBS info:    4805808 (   25472.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     345437 (    1830.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      188.669 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               46.534
  2                0.000               63.789
  3                0.000               54.847
  4                0.000              174.519
  5                0.000              123.945
  6                0.000              113.898
  7                0.000              135.923
  8                0.000               59.774
  9                0.000               27.296
 10                0.000               12.263
 11                0.000               65.598
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14400003     100475          0          0          0          0       1767       1767      95813       1128          0      35602
  1   23251868      63483          0          0          0          0       2988       2988      57507          0          0      21204
  2   17192461    2025458          0          0          0          0       2415       2415     200396    1820232          0      34093
  3   20019932    1908259          0          0          0          0       2815       2815      81941    1820688          0      32715
  4    3059738     405674          0          0          0          0        403        403     100680     304188          0      35774
  5   10595698     355900          0          0          0          0       1322       1322      58122     295134          0      21319
  6   12037952     388178          0          0          0          0       1554       1554     105230     279840          0      34521
  7    9390918     368608          0          0          0          0       1205       1205      81600     284598          0      31165
  8   46854628      72459          0          0          0          0       5872       5872      60715          0          0      21758
  9   43874737      60599          0          0          0          0       5426       5426      49747          0          0      17382
 10   36184433     112868          0          0          0          0       4528       4528     103812          0          0      31745
 11   19113175      82496          0          0          0          0       2408       2408      77680          0          0      28159
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 1641.051s (   19.500 blocks/s,   1.219 Mb/s)
Attachment 5: 210420_0109_Bias.png
210420_0109_Bias.png
Attachment 6: 210420_0026_R34_36_Analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255999603 ( 1309621.8 Hz)
        Other data format:    5920397 (   30287.1 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32728 (     167.4 Hz)
                              WR48-63:      32728 (     167.4 Hz)
                           FEE64 disc:    1044504 (    5343.4 Hz)
                             MBS info:    4810437 (   24608.8 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     327887 (    1677.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      195.476 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               51.325
  2                0.000               68.599
  3                0.000               55.115
  4                0.000              180.174
  5                0.000              140.295
  6                0.000              123.427
  7                0.000              138.446
  8                0.000               55.510
  9                0.000               13.463
 10                0.000               24.357
 11                0.000               74.336
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   14611253      94841          0          0          0          0       1936       1936      89802       1167          0      33511
  1   23423897      61346          0          0          0          0       3006       3006      55334          0          0      20222
  2   17311487    2050104          0          0          0          0       2477       2477     195362    1849788          0      32021
  3   20533060    1944396          0          0          0          0       2862       2862      77493    1861179          0      30741
  4    2942569     381933          0          0          0          0        417        417      93930     287169          0      33667
  5    9776355     338395          0          0          0          0       1239       1239      56110     279807          0      20096
  6   11887130     369619          0          0          0          0       1555       1555     103862     262647          0      32503
  7    9239609     348065          0          0          0          0       1200       1200      76985     268680          0      29470
  8   49507336      72711          0          0          0          0       6129       6129      60453          0          0      21190
  9   44170393      65100          0          0          0          0       5430       5430      54240          0          0      18016
 10   32977762     112783          0          0          0          0       4022       4022     104739          0          0      30280
 11   19618752      81104          0          0          0          0       2455       2455      76194          0          0      26170
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time: 5995.524s (    5.337 blocks/s,   0.334 Mb/s)
Attachment 7: 210420_0220_Stats.png
210420_0220_Stats.png
Attachment 8: 210420_0229_Temp.png
210420_0229_Temp.png
Attachment 9: 210420_0220_Bias.png
210420_0220_Bias.png
Attachment 10: 210420_0225_MergerStats.png
210420_0225_MergerStats.png
Attachment 11: Capture.PNG
Capture.PNG
Attachment 12: 210420_0437_stats.png
210420_0437_stats.png
Attachment 13: 210420_0538_Temp.png
210420_0538_Temp.png
Attachment 14: 210420_0439_Bias.png
210420_0439_Bias.png
Attachment 15: 210420_0624_stats.png
210420_0624_stats.png
Attachment 16: 210420_0624_Temp.png
210420_0624_Temp.png
Attachment 17: 210420_0625_Bias.png
210420_0625_Bias.png
  249   Mon Apr 19 19:19:18 2021 CBMonday 19 April 20:00-24:00 shift
20:19 
Statistics OK (attach 1)
No faults found
Ucesb OK (attach 2)
Temps OK (attach 3)
Bias OK (attach 4)

System-wide checks - as per last check
aida09 fails clock (OK)
aida07 and 09 fail calibration (OK)

		 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd05f : 	 13  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x29 : 	 41  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      0      3      2      2      2      5      2      3      2      3      6   : 35704
aida02 :     19      4      1      1      1      2      2      3      3      3      6   : 36316
aida03 :     12      9      2      1      2      2      2      3      3      3      6   : 36408
aida04 :     16      3      5      4      2      4      2      3      3      3      6   : 36776
aida05 :     41     10     10      3      3      3      3      3      3      3      6   : 37172
aida06 :     25     21      9      3      2      3      1      4      3      3      6   : 37116
aida07 :     10      7      1      2      1      3      2      2      3      3      6   : 35952
aida08 :     33      7      5      1      1      2      2      3      3      3      6   : 36460
aida09 :     12      7      4      2      2      2      2      2      2      4      6   : 36968
aida10 :     16      6      2      1      1      4      3      3      2      3      6   : 35824
aida11 :      2      0      1      2      2      2      2      4      2      3      6   : 35800
aida12 :     15      6      2      0      2      4      2      2      3      3      6   : 36108



20:48 Started compressing *old* runs on Screen 2, Workspace 5
nice -n10 gzip -v R[...]_*

21:00 Stats OK, UCESB OK, No faults

21:07 Go4 spectra erased by remote shifters.

21:15 Local shifters report correlations lost. Beam may be intermittently removed for checks.

21:30 Still no beam. DAQ issues locally. AIDA MBS link had to be restarted due to operator mistake.




21:37 Beam back. Stats OK, UCESB OK, No faults found.

System wide checks:
Clock - aida09 fails

All modules now fail ADC calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd061 : 	 15  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x36 : 	 54  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      3      1      2      4      2      4      2      3      6   : 36104
aida02 :      2      3      0      2      2      4      2      2      3      3      6   : 36064
aida03 :      6      3      3      3      2      2      2      2      4      3      6   : 36928
aida04 :     15      5      8      4      3      3      2      3      3      3      6   : 36772
aida05 :     10      5      8      4      4      4      3      2      3      3      6   : 36688
aida06 :     14     17     10      3      3      3      1      4      3      3      6   : 37120
aida07 :      3      4      1      1      1      2      2      2      3      3      6   : 35740
aida08 :     12      5      7      2      2      4      1      2      3      3      6   : 35976
aida09 :      0      6      2      0      2      2      2      2      2      4      6   : 36816
aida10 :      4      2      5      2      1      4      2      4      2      3      6   : 36080
aida11 :     14      3      2      3      1      2      2      4      2      3      6   : 35856
aida12 :      1      2      0      2      1      4      1      3      3      3      6   : 36244

22:05 Stats OK (attach 5)
Temps OK (attach 6)
Bias OK (attach 7)

System wide checks as before - see above.

22:13 ADC calibration - Only 06,07 and 09 now fail
(TD re-calibrated remotely)

22:53 Beam off to allow access to S4 to reboot VME crate (akash).


22:58 Started compressing *old* runs on Screen 2, Workspace 5
nice -n10 gzip -v R[1-2][0-9]_* R[3][0-2]_*
aka everything *except* R33_*


23:11: Beam is back. Stats and UCESB as before.

23:24: Stats OK (attach 8)
Temps OK (attach 9)
Bias OK (attach 10)
UCESB OK (attach 11)
System-wide checks
Clock: 06 fails
Calibration: 06,07,09 fail

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd066 : 	 20  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd066 : 	 20  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      8      2      2      3      4      3      3      2      3      6   : 35996
aida02 :      8     12      4      1      2      4      3      2      3      3      6   : 36448
aida03 :      2      4      2      4      3      3      1      3      3      3      6   : 36360
aida04 :      7      8      4      4      2      4      3      3      3      3      6   : 37020
aida05 :      3      5      7      4      3      3      3      3      3      3      6   : 36964
aida06 :      2      6      8      2      3      4      2      3      3      3      6   : 36792
aida07 :      8      4      5      2      0      3      1      2      3      4      6   : 37712
aida08 :     26     11      3      1      1      3      1      3      3      3      6   : 36304
aida09 :     22      4      2      2      3      4      1      3      3      3      6   : 36504
aida10 :     21      6      6      2      0      4      2      3      3      3      6   : 36644
aida11 :     22      7      2      3      2      1      2      4      2      3      6   : 35856
aida12 :     16      9      2      2      3      3      2      2      3      3      6   : 36136
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Ucesb1.png
Ucesb1.png
Attachment 3: Temps1.png
Temps1.png
Attachment 4: Bias1.png
Bias1.png
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Temp2.png
Temp2.png
Attachment 7: Bias2.png
Bias2.png
Attachment 8: Stats3.png
Stats3.png
Attachment 9: Temps3.png
Temps3.png
Attachment 10: Bias3.png
Bias3.png
Attachment 11: Ucesb2.png
Ucesb2.png
  248   Mon Apr 19 16:58:29 2021 LJWMonday 19th April 16:00-20:00 Shift
System Checks @ ~18:10 :


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd05a : 	 8  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x23 : 	 35  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      6      4      0      1      5      2      3      2      3      6   : 35636
aida02 :     27      3      2      0      2      3      2      3      3      3      6   : 36516
aida03 :     33      9      2      1      1      3      2      3      3      3      6   : 36556
aida04 :     11      6      5      4      2      4      3      3      3      3      6   : 37036
aida05 :     15      8      7      4      3      4      4      2      3      3      6   : 36908
aida06 :     13      9      7      3      3      4      2      3      3      3      6   : 36876
aida07 :      1      3      3      2      2      3      1      3      3      3      6   : 36236
aida08 :     29      5      2      1      1      2      2      3      3      3      6   : 36380
aida09 :      3      8      2      0      2      3      2      2      2      4      6   : 36972
aida10 :      9      5      3      3      1      4      2      4      2      3      6   : 36124
aida11 :      5      7      3      2      2      2      2      4      2      3      6   : 35900
aida12 :     24      6      3      1      2      3      2      2      3      3      6   : 36064


System Checks @ ~19:55

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd05e : 	 12  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x27 : 	 39  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     18     10      2      2      1      5      2      3      2      3      6   : 35768
aida02 :     21      9      2      1      2      2      2      3      3      3      6   : 36444
aida03 :     13      5      6      1      2      2      2      3      3      3      6   : 36444
aida04 :     30     11      4      3      2      4      3      3      3      3      6   : 37104
aida05 :     10      9      8      4      4      4      4      2      3      3      6   : 36976
aida06 :     22     17     10      3      2      3      1      4      3      3      6   : 37088
aida07 :     20      8      4      1      2      4      1      2      3      3      6   : 35952
aida08 :     14      4      3      0      1      2      2      3      3      3      6   : 36296
aida09 :     24      8      4      0      1      3      3      3      3      3      6   : 36768
aida10 :     19      8      4      1      1      4      2      3      3      3      6   : 36652
aida11 :      6      6      1      2      3      1      2      4      2      3      6   : 35800
aida12 :     15      7      1      1      2      4      1      2      3      3      6   : 35876
Attachment 1: 63.png
63.png
Attachment 2: Screenshot_2021-04-19_Statistics_aidas-gsi(3).png
Screenshot_2021-04-19_Statistics_aidas-gsi(3).png
Attachment 3: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 4: 64.png
64.png
Attachment 5: 65.png
65.png
Attachment 6: 66.png
66.png
  246   Mon Apr 19 13:27:08 2021 RDPMonday 19th April 12 noon shift
12.30 There appear to be fewer HEC events in aida09 cf. aida11 and upstream DSSSDs
      Decrease aida09 asic #1-4 HEC fast comparator 0x2 -> 0x1
      Online/nearline analyses of daat files indicates that this does not resolve the issue

Some timestamp issues 

MERGE Data Link (23725): bad timestamp  11 3 0x8b620008 0x0287be94 0x00003dd78287be94 0x16773dd78287be94 0x16773dd78287beb2
MERGE Data Link (23725): bad timestamp  11 3 0x8b610040 0x073302ce 0x00003df3e73302ce 0x16773df3e73302ce 0x16773df3e73302ec
MERGE Data Link (23725): bad timestamp  11 3 0x8b610020 0x0a54b794 0x00003e4e4a54b794 0x16773e4e4a54b794 0x16773e4e4a54b7b2
MERGE Data Link (23725): bad timestamp  11 3 0x8b610020 0x052da21e 0x00003f68f52da21e 0x16773f68f52da21e 0x16773f68f52da232
MERGE Data Link (23725): bad timestamp  11 3 0x8b620008 0x0061489e 0x000040aa8061489e 0x167740aa8061489e 0x167740aa806148a8
MERGE Data Link (23725): bad timestamp  11 3 0x8b610008 0x04029d5e 0x00004169c4029d5e 0x16774169c4029d5e 0x16774169c4029d68
MERGE Data Link (23725): bad timestamp  11 3 0x8b630004 0x08dfcb14 0x0000417fd8dfcb14 0x1677417fd8dfcb14 0x1677417fd8dfcb28
MERGE Data Link (23725): bad timestamp  11 3 0x8b630020 0x08dfcb1e 0x0000417fd8dfcb1e 0x1677417fd8dfcb1e 0x1677417fd8dfcb28

13.50 Increased DSSSD# 3 bias from -100V

      -100V -6.325uA
      -110V -6.500uA
      -120V -6.800uA



       
      
Attachment 1: 61.png
61.png
Attachment 2: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 3: Screenshot_2021-04-19_Statistics_aidas-gsi(2).png
Screenshot_2021-04-19_Statistics_aidas-gsi(2).png
  244   Mon Apr 19 06:46:15 2021 MSMonday 19 April 07:47-12:00 2021

07:47

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

    
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c46 :      74  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

        
             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x35 :      43  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     18     10      7      1      3      2      4      2      3      6   : 36380
aida02 :     11     15      5      3      2      3      2      3      1      4      6   : 36692
aida03 :     15     11     13      3      4      3      3      3      2      3      6   : 36164
aida04 :     22     14     20      5      1      3      4      3      2      3      6   : 36456
aida05 :      7      7      5      2      1      2      3      4      2      3      6   : 36132
aida06 :     21     14      6      2      3      3      1      2      2      4      6   : 37028
aida07 :     11      8      7      4      2      3      2      2      4      3      6   : 37212
aida08 :     18     10      7      5      3      1      4      3      2      3      6   : 36072
aida09 :     24     23     18      3      2      3      2      4      2      3      6   : 36504
aida10 :      6     19      5      2      2      2      2      3      2      3      6   : 35520
aida11 :     22     15      7      2      1      3      2      3      2      3      6   : 35648
aida12 :     35     13      8      2      5      3      1      3      3      3      6   : 36724

 

At 8:30 DAQ and Temperature Scan stopped working properly.

 

 

11:00

It was noted that at the time that AIDA dropped out ~8:30 the beam also dropped out. It seems a bit of a coincidence

TD and MS powercycled the FEEs after the first crash and restarted MIDAS but not the merger. This recovered the FEEs but did not re-establish the links between the FEEs and the Merger.

Another powercycle was performed this time with a full reset of the merger and the links were restored. Upon restoring there was a large amount of noise in across all FEEs on average a 50% increase across all FEEs but in DSSD a factor of 4-6 increase was common.

In the waveforms large 100kHz transiets could be seen - attachment 4

A further power-cycle was performed by OH. The rates following this powercycle are better than just before the power cycle but have not recovered to pre-glitch levels.

The waveforms here are much improved though - attachment 5 and 6

 

 

11:45

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

 

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      1      2      2      1      2      4      2      4      2      3      7   : 40148
aida02 :      5      4      2      2      1      2      2      3      2      3      7   : 39380
aida03 :      3      2      1      1      1      3      1      4      2      3      7   : 39692
aida04 :      4      2      1      0      1      3      3      3      2      3      7   : 39664
aida05 :      2      7      2      0      2      2      3      4      2      3      7   : 40160
aida06 :      2      3      2      2      2      2      1      3      3      3      7   : 40192
aida07 :     23      7      2      0      0      3      2      4      2      3      7   : 39988
aida08 :     23     11      4      0      1      3      1      4      2      3      7   : 39860
aida09 :      1      3      1      1      2      4      1      3      1      4      7   : 40396
aida10 :      1      3      3      1      1      4      1      2      3      3      7   : 39852
aida11 :      3      0      2      3      1      2      3      2      2      3      7   : 39116
aida12 :      5      8      0      3      1      4      2      3      2      3      7   : 39668

 

 

 

 

 

 

 

 

Attachment 1: Screenshot_2021-04-19_Statistics_aidas-gsi.png
Screenshot_2021-04-19_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 42.png
42.png
Attachment 4: Screenshot_2021-04-19_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-19_Spectrum_Browser_aidas-gsi(1).png
Attachment 5: 210319_1054_Layout7.png
210319_1054_Layout7.png
Attachment 6: 210419_1054_Layout8.png
210419_1054_Layout8.png
Attachment 7: Screenshot_2021-04-19_Statistics_aidas-gsi(1).png
Screenshot_2021-04-19_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 43.png
43.png
  243   Mon Apr 19 01:08:40 2021 MAMonday 19th April 00:00-08:00

03:09   General check

Rates, temptature, voltages are OK attached 1, 2, 3, 4

 

****Clock Ckeck******

OK
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

******ADC check ******
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

****** White Rabbit check******

     Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c2c :      48  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

*****FPGA check ******


             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x10 :      6  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

*****Memory check*****


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      6     12     11      7      2      3      2      4      2      3      6   : 36360
aida02 :     21     15      7      3      2      3      1      4      2      3      6   : 35996
aida03 :      9     10     10      2      4      3      3      3      2      3      6   : 36052
aida04 :     17     17     17      4      0      2      3      4      2      3      6   : 36444
aida05 :     25      7      3      1      1      1      2      3      2      4      6   : 37292
aida06 :     29     15      7      2      3      4      1      3      3      3      6   : 36700
aida07 :     17     17      6      5      2      3      2      3      3      3      6   : 36812
aida08 :     22      6     10      5      3      1      3      4      2      3      6   : 36360
aida09 :     18     28     14      4      2      3      3      3      2      3      6   : 36232
aida10 :     26     16      4      3      3      1      3      4      2      3      6   : 36296
aida11 :     16     10      4      0      3      3      1      4      2      3      6   : 35856
aida12 :     19     14      4      2      4      4      1      3      3      3      6   : 36668

 

04:15      General Check

Rates, Tempratures, Voltages are ok, attached 5,6,7,8

******Clock Check*******

OK

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

******ADC******

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

******White Rabbit check******


         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c39 :      61  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


******FPGA Check******

     Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x11 :      7  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


****** Memorey check******


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     27     21     10      7      1      3      2      4      2      3      6   : 36436
aida02 :     28     21      3      3      1      3      1      4      2      3      6   : 35944
aida03 :     15      8     11      3      3      3      3      3      2      3      6   : 36044
aida04 :     25     19     14      4      1      3      4      3      2      3      6   : 36380
aida05 :     12      6      5      1      0      2      2      3      2      4      6   : 37328
aida06 :     29     21      4      2      3      4      1      3      3      3      6   : 36700
aida07 :     19     14      7      6      2      3      2      2      3      3      6   : 36332
aida08 :     11      8      8      6      3      1      3      4      2      3      6   : 36332
aida09 :     16     25     11      3      1      2      2      4      2      3      6   : 36184
aida10 :     14     13      4      2      3      2      2      4      2      3      6   : 36064
aida11 :      4      9      4      0      3      3      1      4      2      3      6   : 35800
aida12 :     38     17      4      2      4      4      1      3      3      3      6   : 36768

 

 

 

 

Attachment 1: 210419_0302_Voltages.png
210419_0302_Voltages.png
Attachment 2: 210419_0303_Stats.png
210419_0303_Stats.png
Attachment 3: 210419_0303_Temp.png
210419_0303_Temp.png
Attachment 4: 210419_0406_ucesb.png
210419_0406_ucesb.png
Attachment 5: 210419_0402_Stats.png
210419_0402_Stats.png
Attachment 6: 210419_0402_Voltages.png
210419_0402_Voltages.png
Attachment 7: 210419_0406_ucesb.png
210419_0406_ucesb.png
Attachment 8: 210419_0414_ucesb.png
210419_0414_ucesb.png
  242   Sun Apr 18 21:43:01 2021 ML-OHsystem wide checks

New System wide check 10pm CET

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC calibration check:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

   
         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c1d :      33  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  


White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp error checks:

             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0xf :      5  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Memory from FEE64:
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     22     19      6      4      2      3      2      4      2      3      6   : 36304
aida02 :     23      9      3      3      1      3      1      4      2      3      6   : 35828
aida03 :     15      5      6      3      2      3      3      3      2      3      6   : 35876
aida04 :     20      8     18      1      1      3      4      3      2      3      6   : 36240
aida05 :     21      7      1      2      1      2      4      3      2      3      6   : 35868
aida06 :     18      8      4      1      3      4      1      3      3      3      6   : 36520
aida07 :      4      9      3      4      2      3      2      2      3      3      6   : 36104
aida08 :     15      5      2      4      2      1      3      4      2      3      6   : 36100
aida09 :      2     23      8      1      2      3      2      4      2      3      6   : 36192
aida10 :     10      7      8      5      2      1      3      3      2      3      6   : 35712
aida11 :      1      9      5      1      2      3      1      4      2      3      6   : 35772
aida12 :     14      8      6      3      3      4      1      3      3      3      6   : 36600

Attachment 1: 18-04-2021_10pm_HV.png
18-04-2021_10pm_HV.png
Attachment 2: 18-04-2021_10pm_StatGoodEvt.png
18-04-2021_10pm_StatGoodEvt.png
Attachment 3: 18-04-2021_10pm_PAUSE.png
18-04-2021_10pm_PAUSE.png
Attachment 4: 18-04-2021_10pm_RESUME.png
18-04-2021_10pm_RESUME.png
Attachment 5: 18-04-2021_10pm_ADCdataItem.png
18-04-2021_10pm_ADCdataItem.png
Attachment 6: 18-04-2021_10pm_DISC.png
18-04-2021_10pm_DISC.png
Attachment 7: 18-04-2021_10pm_correlations.png
18-04-2021_10pm_correlations.png
Attachment 8: 18-04-2021_10pm_SpecRate.png
18-04-2021_10pm_SpecRate.png
Attachment 9: 18-04-2021_10pm_Spec1-8H.png
18-04-2021_10pm_Spec1-8H.png
Attachment 10: 18-04-2021_10pm_Spec1-8HlayoutID6.png
18-04-2021_10pm_Spec1-8HlayoutID6.png
Attachment 11: 18-04-2021_10pm_Spec1-8L.png
18-04-2021_10pm_Spec1-8L.png
Attachment 12: 18-04-2021_10pm_Spec1-8L_layoutid4.png
18-04-2021_10pm_Spec1-8L_layoutid4.png
Attachment 13: 18-04-2021_10pm_Spec1-8W.png
18-04-2021_10pm_Spec1-8W.png
Attachment 14: 18-04-2021_10pm_SpecLayoutID8.png
18-04-2021_10pm_SpecLayoutID8.png
Attachment 15: 18-04-2021_10pm_WR28-47.png
18-04-2021_10pm_WR28-47.png
Attachment 16: 18-04-2021_10pm_WR48-63.png
18-04-2021_10pm_WR48-63.png
Attachment 17: 18-04-2021_10pm_temp.png
18-04-2021_10pm_temp.png
Attachment 18: 18-04-2021_10pm_ucesb_monitor.png
18-04-2021_10pm_ucesb_monitor.png
Attachment 19: 18-04-2021_10pm-AIDA_Alerting_-_Grafana.png
18-04-2021_10pm-AIDA_Alerting_-_Grafana.png
Attachment 20: 18-04-2021_10pm_NewMerger.png
18-04-2021_10pm_NewMerger.png
  241   Sun Apr 18 19:05:33 2021 ML-OHAll spectra reset

 All spectra reset at 8pm CET.

 

  240   Sun Apr 18 17:27:44 2021 ML-OHsystem wide checks

system wide checks at 6pm CET

##################################################

Difference noted from previous shift:

FPGS timestamp error 10 passed and 2 failed

##################################################


Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

ADC Calibration:

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit checks:

         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c11 :      21  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


FPGA timestamp errors checks:

             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0xc :      2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Memory information fro FEE64:

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     27     16      4      2      2      2      1      3      3      3      6   : 36332
aida02 :     19      8      9      3      2      2      2      4      2      3      6   : 36092
aida03 :     23     10      8      6      4      4      3      3      2      3      6   : 36332
aida04 :     27      4      8      0      1      3      4      3      2      3      6   : 36044
aida05 :     21      6      5      4      4      2      2      2      2      4      6   : 37204
aida06 :     23     11      5      0      3      4      1      3      3      3      6   : 36548
aida07 :     14      9      5      2      3      3      2      2      4      3      6   : 37200
aida08 :     26      8      7      2      2      1      4      3      2      3      6   : 35928
aida09 :      2      3      4      1      1      3      3      4      2      3      6   : 36160
aida10 :     16      9      2      5      1      1      3      4      2      3      6   : 36104
aida11 :      7      5      0      0      3      3      1      4      2      3      6   : 35716
aida12 :      9      7      3      2      4      4      1      3      3      3      6   : 36556

 

 

 

Attachment 1: 18-04-2021_6pm_CAENHV.png
18-04-2021_6pm_CAENHV.png
Attachment 2: 18-04-2021_6pm_StatGoodEvent.png
18-04-2021_6pm_StatGoodEvent.png
Attachment 3: 18-04-2021_6pm_ADCdataItem.png
18-04-2021_6pm_ADCdataItem.png
Attachment 4: 18-04-2021_6pm_WR28-47#4.png
18-04-2021_6pm_WR28-47#4.png
Attachment 5: 18-04-2021_6pm_WR48-63#5.png
18-04-2021_6pm_WR48-63#5.png
Attachment 6: 18-04-2021_6pm_PauseInfo.png
18-04-2021_6pm_PauseInfo.png
Attachment 7: 18-04-2021_6pm_ResumeInfo.png
18-04-2021_6pm_ResumeInfo.png
Attachment 8: 18-04-2021_6pm_DiscInfo.png
18-04-2021_6pm_DiscInfo.png
Attachment 9: 18-04-2021_6pm_Correlation#8.png
18-04-2021_6pm_Correlation#8.png
Attachment 10: 18-04-2021_6pm_ucesb.png
18-04-2021_6pm_ucesb.png
Attachment 11: 18-04-2021_645pm_tempScan.png
18-04-2021_645pm_tempScan.png
Attachment 12: 18-04-2021_645pm_Spec1-8L.png
18-04-2021_645pm_Spec1-8L.png
Attachment 13: 18-04-2021_7pm_Spec1-8L_2.png
18-04-2021_7pm_Spec1-8L_2.png
Attachment 14: 18-04-2021_7pm_Spec1-8_layoutId4.png
18-04-2021_7pm_Spec1-8_layoutId4.png
Attachment 15: 18-04-2021_7pm_Spec1-8H.png
18-04-2021_7pm_Spec1-8H.png
Attachment 16: 18-04-2021_645pm_SpecRate.png
18-04-2021_645pm_SpecRate.png
Attachment 17: 18-04-2021_730pm_AIDA_LC.png
18-04-2021_730pm_AIDA_LC.png
  239   Sun Apr 18 11:18:21 2021 TDSunday 18 April 12.00-16.00
12.18 DAQ continues file NULL/R30_490

      ASIC settings 2019Dec19-16.19.51
      slow comparator 0x64 -> 0xa

      BNC PB-5
      amplitude 2V
      attenuation 1x
      decay time 1ms
      frequency 2Hz

12.22 all histograms zero'd

c. 12.30 219Rn setting

12.35 analysis of R30_490 - see attachment 1

13.15 system wide checks

Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida01 fault 	 0x7686 : 	 0x768a : 	 4  
aida02 fault 	 0x941d : 	 0x9421 : 	 4  
aida03 fault 	 0x7cd7 : 	 0x7cdb : 	 4  
aida04 fault 	 0xb86d : 	 0xb871 : 	 4  
aida05 fault 	 0x1a59 : 	 0x1a5f : 	 6  
aida06 fault 	 0x4f45 : 	 0x4f4b : 	 6  
aida07 fault 	 0x3bfc : 	 0x3c04 : 	 8  
aida08 fault 	 0xc7ce : 	 0xc7d4 : 	 6  
aida09 fault 	 0xb33b : 	 0xb33c : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0xa : 	 0xb : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23     10      3      1      1      2      1      3      3      3      6   : 36156
aida02 :      2     11      7      4      2      2      2      4      2      3      6   : 36048
aida03 :     21      8      6      4      5      4      3      3      2      3      6   : 36276
aida04 :     19     11     14      3      2      3      2      4      2      3      6   : 36324
aida05 :     25      4      5      4      4      2      1      3      2      4      6   : 37460
aida06 :     13      9     15      1      3      5      1      3      3      3      6   : 36812
aida07 :     19     10      6      1      3      3      2      2      4      3      6   : 37212
aida08 :     23      6      4      2      1      1      3      4      2      3      6   : 36044
aida09 :     11      5      5      1      2      4      3      3      2      3      6   : 35908
aida10 :     14      5      6      5      3      1      3      3      2      3      6   : 35744
aida11 :     16      5      3      0      3      3      1      4      2      3      6   : 35800
aida12 :     12     10      0      2      4      4      1      3      3      3      6   : 36544

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

*no faults detected by ~/oh/OptionsCheck.py

13.10 Detector biases & leakage currents OK - see attachment 2
      Statistics - good events, adc data, disc data, info code 4 & 5, pause, resume, correlation info - attachments 3-12
      per FEE64 rate spectra - attachments 13-14
      1.8.L spectra - attachments 15-18

       pulser peak widths
       1 80
       2 113
       3 308
       4 105
       5 50
       6 91
       7 61
       8 90
       9 200
       10 133
       11 187
       12 107
  
      1.8.H spectra - attachments 19-20
      1.8.W spectra - attachments 21-22
      ucesb - attachment 23
       implant rates DSSSD#1-3 to c. 1kHz
      grafana - AIDA DSSSD leakage currents for previous 2 days - attachment 24
      merger & merger/tape server/mbs data tranfser - attachment 25
       no recent merger warning/error messages

13.30 225At setting
Attachment 1: R30_490
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256516272 ( 1223332.2 Hz)
        Other data format:    5403728 (   25770.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:      32781 (     156.3 Hz)
                              WR48-63:      32781 (     156.3 Hz)
                           FEE64 disc:     828389 (    3950.6 Hz)
                             MBS info:    4509777 (   21507.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     263493 (    1256.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      209.687 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000               76.433
  2                0.000               87.604
  3                0.000               78.795
  4                0.000              195.707
  5                0.000              158.854
  6                0.000              193.755
  7                0.000              152.656
  8                0.000               23.126
  9                0.000               23.243
 10                0.000               11.178
 11                0.000              108.457
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9851554      72944          0          0          0          0       1261       1261      69168       1254          0      26448
  1   21158309      48956          0          0          0          0       2701       2701      43554          0          0      16374
  2   17027137    1992411          0          0          0          0       2376       2376     156273    1831386          0      25239
  3   18790255    1894647          0          0          0          0       2567       2567      58490    1831023          0      23746
  4    2269151     296174          0          0          0          0        339        339      74660     220836          0      26646
  5    8373135     260125          0          0          0          0       1152       1152      43783     214038          0      16230
  6    4256540     286725          0          0          0          0        532        532      81589     204072          0      25586
  7    9526319     267328          0          0          0          0       1246       1246      57668     207168          0      22643
  8   54059298      76554          0          0          0          0       6619       6619      63316          0          0      22746
  9   49446691      53505          0          0          0          0       6256       6256      40993          0          0      14045
 10   45199495      92635          0          0          0          0       5711       5711      81213          0          0      23246
 11   16558388      61724          0          0          0          0       2021       2021      57682          0          0      20544
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:44645.059s (    0.717 blocks/s,   0.045 Mb/s)
Attachment 2: 50.png
50.png
Attachment 3: Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 5: Screenshot_2021-04-18_Statistics_aidas-gsi(2).png
Screenshot_2021-04-18_Statistics_aidas-gsi(2).png
Attachment 6: Screenshot_2021-04-18_Statistics_aidas-gsi(3).png
Screenshot_2021-04-18_Statistics_aidas-gsi(3).png
Attachment 7: Screenshot_2021-04-18_Statistics_aidas-gsi(4).png
Screenshot_2021-04-18_Statistics_aidas-gsi(4).png
Attachment 8: Screenshot_2021-04-18_Statistics_aidas-gsi(5).png
Screenshot_2021-04-18_Statistics_aidas-gsi(5).png
Attachment 9: Screenshot_2021-04-18_Statistics_aidas-gsi(6).png
Screenshot_2021-04-18_Statistics_aidas-gsi(6).png
Attachment 10: Screenshot_2021-04-18_Statistics_aidas-gsi(7).png
Screenshot_2021-04-18_Statistics_aidas-gsi(7).png
Attachment 11: Screenshot_2021-04-18_Statistics_aidas-gsi(8).png
Screenshot_2021-04-18_Statistics_aidas-gsi(8).png
Attachment 12: Screenshot_2021-04-18_Statistics_aidas-gsi(9).png
Screenshot_2021-04-18_Statistics_aidas-gsi(9).png
Attachment 13: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(1).png
Attachment 15: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(2).png
Attachment 16: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(3).png
Attachment 17: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(4).png
Attachment 18: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(5).png
Attachment 19: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(6).png
Attachment 20: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(7).png
Attachment 21: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(8).png
Attachment 22: Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-04-18_Spectrum_Browser_aidas-gsi(9).png
Attachment 23: Screenshot_2021-04-18_ucesb.png
Screenshot_2021-04-18_ucesb.png
Attachment 24: Screenshot_2021-04-18_AIDA_Alerting_-_Grafana.png
Screenshot_2021-04-18_AIDA_Alerting_-_Grafana.png
Attachment 25: 51.png
51.png
  238   Sun Apr 18 07:30:20 2021 DJ, TDSunday 18th April 08:00-12:00

---- 09-26
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

---


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

--


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

-


         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bfc :      47  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

 

08.45 TD resets baseline for WR and FPGA errors

 

Attachment 1: Screenshot_2021-04-18_Statistics_aidas-gsi.png
Screenshot_2021-04-18_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 52.png
52.png
Attachment 4: 53.png
53.png
Attachment 5: Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-18_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 6: 54.png
54.png
Attachment 7: Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
Screenshot_2021-04-18_Statistics_aidas-gsi(1).png
  237   Sun Apr 18 00:57:36 2021 BA, MASanday 18 April 00.00-08.00

02:01 Beam has stopped at 01:37 and returend at 01:43 for few min and then stopped again and not knowen how it will take until it is back

AIDA scalers attached 1

statistic attached 2

temretuer attached 3

bias attached 4

Clock check ok

ADC check :

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


         Base         Current     Difference
aida05 fault      0x1a52 :      0x1a55 :      3  
aida06 fault      0x4f3e :      0x4f41 :      3  
aida07 fault      0x3bcd :      0x3bea :      29  
aida08 fault      0xc7c7 :      0xc7ca :      3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

   
             Base         Current         Difference
aida09 fault      0x0 :      0x1 :      1  
aida12 fault      0x0 :      0x9 :      9  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     32      8      3      1      0      3      1      3      3      3      6   : 36240
aida02 :     11      7      9      3      1      2      2      4      2      3      6   : 35988
aida03 :     32      4     13      5      5      4      3      3      2      3      6   : 36432
aida04 :     26      7      6      1      2      3      2      4      2      3      6   : 36128
aida05 :     18      8      8      7      4      2      2      2      2      4      6   : 37352
aida06 :     24     11      3      1      2      5      1      3      3      3      6   : 36616
aida07 :     14      5      3      3      3      2      3      2      4      3      6   : 37296
aida08 :     21      7      1      5      0      1      2      3      3      3      6   : 36284
aida09 :      0      7      1      1      1      3      2      2      3      3      6   : 35880
aida10 :     22      5      3      4      1      2      2      2      3      3      6   : 35952
aida11 :      4      3      1      1      2      2      3      3      2      3      6   : 35544
aida12 :     22     10      5      4      4      4      1      3      3      3      6   : 36728

 

02:19  beam is back

03:58 The beam has not been stable yet

           The rate reach 1.5 kHz, they will contact FRS team to lower the intensity of the beam

 

AIDA scalers attached 8

statistic attached 7

temretuer attached 6

bias attached 5

 

   
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

 

 

 

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

   
         Base         Current     Difference
aida05 fault      0x1a52 :      0x1a55 :      3  
aida06 fault      0x4f3e :      0x4f41 :      3  
aida07 fault      0x3bcd :      0x3beb :      30  
aida08 fault      0xc7c7 :      0xc7ca :      3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

   
             Base         Current         Difference
aida09 fault      0x0 :      0x1 :      1  
aida12 fault      0x0 :      0x9 :      9  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

   
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     21      8      6      2      1      2      1      3      3      3      6   : 36212
aida02 :     24     14     14      2      1      2      2      4      2      3      6   : 36144
aida03 :     31      5     10      5      5      4      2      3      2      3      6   : 36132
aida04 :     12     11     12      2      2      2      3      4      2      3      6   : 36360
aida05 :     23      8      5      7      4      2      2      2      2      4      6   : 37324
aida06 :     21     14     14      1      3      5      1      3      3      3      6   : 36868
aida07 :     15      9      5      0      3      2      3      2      4      3      6   : 37268
aida08 :     21     11     10      3      0      1      2      3      3      3      6   : 36396
aida09 :      5      7      5      1      1      3      1      3      3      3      6   : 36220
aida10 :     15     13     12      3      2      1      2      2      3      3      6   : 36036
aida11 :     13     10      1      0      2      2      2      4      2      3      6   : 35860
aida12 :     29      5      4      5      5      3      1      3      3      3      6   : 36668

 

05: 07  The rate was a bout 1500 and 2000, we contacted Oscar he said (if it's just bursts it should be ok), so they decided to do nothing.

 

06:26 

AIDA scalers attached 9

statistic attached 10

temretuer attached 11

bias attached 12

 

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

         Base         Current     Difference
aida01 fault      0x7685 :      0x7686 :      1  
aida02 fault      0x941c :      0x941d :      1  
aida03 fault      0x7cd6 :      0x7cd7 :      1  
aida04 fault      0xb86c :      0xb86d :      1  
aida05 fault      0x1a52 :      0x1a59 :      7  
aida06 fault      0x4f3e :      0x4f45 :      7  
aida07 fault      0x3bcd :      0x3bf9 :      44  
aida08 fault      0xc7c7 :      0xc7ce :      7  
aida09 fault      0xb33a :      0xb33b :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

             Base         Current         Difference
aida09 fault      0x0 :      0x1 :      1  
aida12 fault      0x0 :      0xa :      10  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     17     12      6      2      1      2      1      3      3      3      6   : 36228
aida02 :     19     11     12      3      2      2      2      4      2      3      6   : 36164
aida03 :     25      8     12      5      4      4      3      3      2      3      6   : 36356
aida04 :     22     19     11      3      1      2      3      4      2      3      6   : 36416
aida05 :     35      6      5      6      3      2      2      2      3      3      6   : 36236
aida06 :     12     11     11      2      3      4      1      3      3      3      6   : 36664
aida07 :     18      6      2      1      3      2      3      2      3      3      6   : 36216
aida08 :     27     10      8      3      0      1      2      3      3      3      6   : 36380
aida09 :     18      6      6      1      0      2      2      2      3      3      6   : 35832
aida10 :      3     14     10      3      1      1      2      3      3      3      6   : 36412
aida11 :      1      3      2      0      2      2      2      4      2      3      6   : 35772
aida12 :      0      5      6      3      3      4      1      3      3      3      6   : 36520

 

07:57  The beam stopped and they said there is a water leak !

 

 

 

 

 

Attachment 1: 20210418_0154.png
20210418_0154.png
Attachment 2: 20210418_0152_rate.png
20210418_0152_rate.png
Attachment 3: 20210418_0150_tem.png
20210418_0150_tem.png
Attachment 4: 20210418_0150_bias.png
20210418_0150_bias.png
Attachment 5: 20210418_0407.png
20210418_0407.png
Attachment 6: 20210418_0409.png
20210418_0409.png
Attachment 7: 20210418_0411.png
20210418_0411.png
Attachment 8: 20210418_0413.png
20210418_0413.png
Attachment 9: 18042021_0722_AIDA.png
18042021_0722_AIDA.png
Attachment 10: 18042021_0615_rate.png
18042021_0615_rate.png
Attachment 11: 18042021_0616_temp.png
18042021_0616_temp.png
Attachment 12: 18042021_0620_bias.png
18042021_0620_bias.png
  236   Sat Apr 17 19:39:56 2021 DJ, TDSaturday 17 April 20.00-00.00
19.37 per FEE64 rate spectra - attachments 1 & 2
      1.8.L spectra - attachments 3 & 4
      1.8.H spectra - attachments 5-8
      1.8.W spectra - attachments 9 & 10

20.42 DAQ contrinues file NULL/R30_233



		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a55 : 	 3  
aida06 fault 	 0x4f3e : 	 0x4f41 : 	 3  
aida07 fault 	 0x3bcd : 	 0x3bd7 : 	 10  
aida08 fault 	 0xc7c7 : 	 0xc7ca : 	 3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x4 : 	 4  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

-------
23:48
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a55 : 	 3  
aida06 fault 	 0x4f3e : 	 0x4f41 : 	 3  
aida07 fault 	 0x3bcd : 	 0x3bde : 	 17  
aida08 fault 	 0xc7c7 : 	 0xc7ca : 	 3  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x7 : 	 7  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     26      5      6      2      1      3      1      3      3      3      6   : 36336
aida02 :      7      9     10      3      2      2      2      4      2      3      6   : 36068
aida03 :     26      7      8      4      3      4      2      4      2      3      6   : 36448
aida04 :     18      8      2      1      2      2      3      4      2      3      6   : 36168
aida05 :     24      7      9      4      3      1      3      2      2      4      6   : 37352
aida06 :     17     10      3      1      3      5      1      3      3      3      6   : 36644
aida07 :      7     11      2      2      3      2      3      2      4      3      6   : 37268
aida08 :     23      4      1      5      1      1      2      3      3      3      6   : 36332
aida09 :     14      6      3      2      1      3      2      3      3      3      6   : 36504
aida10 :      2      2      2      2      1      2      2      2      3      3      6   : 35768
aida11 :      2      2      1      2      2      2      2      4      2      3      6   : 35816
aida12 :     26      8      5      5      4      3      1      3      3      3      6   : 36632
----
Attachment 1: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(4).png
Attachment 6: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(5).png
Attachment 7: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(6).png
Attachment 8: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(7).png
Attachment 9: Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-17_Spectrum_Browser_aidas-gsi(8).png
Attachment 10: Screenshot_2021-04-17_Statistics_aidas-gsi(4).png
Screenshot_2021-04-17_Statistics_aidas-gsi(4).png
Attachment 11: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 12: 50.png
50.png
Attachment 13: 51.png
51.png
  235   Sat Apr 17 15:29:27 2021 MSSaturday 17 April 16:00-20:00 2021
16:21


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021




18:29
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable



FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



	
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bd1 : 	 4  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last




	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     21      3      4      4      1      3      1      3      3      3      6   : 36332
aida02 :      4      3     13      2      2      3      2      4      2      3      6   : 36152
aida03 :     12      7      5      2      2      3      2      4      2      3      6   : 36088
aida04 :     17      6      9      5      1      1      2      3      3      3      6   : 36452
aida05 :     26      8      0      3      1      1      2      3      3      3      6   : 36296
aida06 :     22      6      9      4      3      4      2      3      3      3      6   : 36952
aida07 :     17      7      3      1      2      3      3      2      3      3      6   : 36300
aida08 :     24      5      4      4      1      1      2      3      3      3      6   : 36360
aida09 :     15     11      3      2      1      3      1      3      3      3      6   : 36292
aida10 :      2      3      5      2      1      2      2      2      3      3      6   : 35824
aida11 :      2      2      0      0      3      2      2      4      2      3      6   : 35800
aida12 :     18     15      7      5      4      3      1      3      3      3      6   : 36688














	
Attachment 1: 40.png
40.png
Attachment 2: Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
Attachment 3: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 6: 41.png
41.png
  234   Sat Apr 17 11:27:38 2021 JS, TDSaturday 17th April 12:00-16:00
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bce : 	 1  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

13:55 CEST

Statistics : ok elog:234/4
Temp : ok  elog:234/5
Bias : ok  elog:234/6
ucesb : ok
DB: No faults found

ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and 
Control browser page to rerun calibration for that module

White Rabbit Check:
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

FPGA check:
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

14:05 no beam
14:10 beam back
14:15 no beam, beam current being optimised, going to thicker degrader when beam returns


14:37 CEST

Statistics : ok
Temp : ok
Bias : ok   (ch3 now over 6uA)
ucesb : ok
DB: No faults found

15:03 CEST

Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found

15:31 CEST

Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found

15:50

Statistics : ok elog:234/7
Temp : ok  elog:234/8
Bias : ok  elog:234/9
ucesb : ok elog:234/10
DB: No faults found

ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit Check:
		 Base 		Current 	Difference
aida05 fault 	 0x1a52 : 	 0x1a53 : 	 1  
aida06 fault 	 0x4f3e : 	 0x4f3f : 	 1  
aida07 fault 	 0x3bcd : 	 0x3bcf : 	 2  
aida08 fault 	 0xc7c7 : 	 0xc7c8 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA check:
			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Attachment 1: 30.png
30.png
Attachment 2: Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
Attachment 3: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 4: Screenshot_2021-04-17_Statistics_aidas-gsi.png
Screenshot_2021-04-17_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: 31.png
31.png
Attachment 7: Screenshot_2021-04-17_Statistics_aidas-gsi(1).png
Screenshot_2021-04-17_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 32.png
32.png
Attachment 10: 33.png
33.png
  233   Sat Apr 17 07:28:02 2021 LPG, TDSaturday 08:00 - 12:00
08:30 CEST

HV and leakage currents: elog:233/1
Detector rates: elog:233/2
Temperatures: elog:233/3

WR status:


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



FPGA status:


             Base         Current         Difference
aida12 fault      0x0 :      0x1 :      1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last





09:20 CEST

Stats: ok!
DB: No faults found
ucesb: ok!


09:40 CEST

Beam will be stopped in order to increase intensity. Expected to be around 3-4 hours.
For now, we still get implants when Beam spill is on. Seems to be fluctuating as they play around.

10:00 CEST

Still getting beam, it is fluctuating in intensity

Stats: ok!
DB: No faults found
ucesb: ok!

HV and leakage currents: elog:233/4
Detector rates: elog:233/5
Temperatures: elog:233/6

Clock check: ok!
ADC calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
WR check: ok!
FPGA check:
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory check: ok!


10:40 CEST

Stats: ok!
DB: No faults found
ucesb: ok!

11:00 CEST

Stats: ok!
DB: No faults found
ucesb: ok!


11:30 CEST

Stats: ok!
DB: No faults found
ucesb: ok!

12:00 BST

Stats: ok!
DB: No faults found
ucesb: ok!

HV and leakage currents: elog:233/7
Detector rates: elog:233/8
Temperatures: elog:233/9


Clock check: ok!
ADC calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
WR check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bce : 1
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
Memory check: ok!
Attachment 1: 20.png
20.png
Attachment 2: Screenshot_2021-04-17_Statistics_aidas-gsi.png
Screenshot_2021-04-17_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: 21.png
21.png
Attachment 5: Screenshot_2021-04-17_Statistics_aidas-gsi(1).png
Screenshot_2021-04-17_Statistics_aidas-gsi(1).png
Attachment 6: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 7: 22.png
22.png
Attachment 8: Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
Attachment 9: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
  232   Fri Apr 16 23:13:57 2021 CA, MA, BASaturday 16th 00:00 - 08:00
00:05 DESPEC tuning beam, changing spill length

00:21 DAQ continues ok
      Merger/TapeServer ok

00:25 still no beam in AIDA

00:30 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 1

00:52 beam in AIDA - rates spectra - attachment 2

01:05 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 3

      beam is off again

01:30 beam back

01:32 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 4

01:37 beam off - FRS increasing spill length to 5s

01:48 beam back with new spill setting - 5s on spill, 2s off spill

02:07 system wide checks ok except;

WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb42 : 	 8  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA check :

			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:10 FEE64 Temps ok - attachment 5

02:20 good event statistics ok - attachment 6

02:27 detector bias / leakage currents ok - attachment 7

02:30 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok

      beam off

03:00 stats/ucesb/database checks ok

03:30 stats/ucesb/database checks ok

      still no beam in AIDA

05:11 system is checked, beam is back with low intensity 

05:13 FEE64 Temps ok - attachment 8

05:13 good event statistics ok - attachment 9

05:13 detector bias / leakage currents ok - attachment 10

05:33 stats/ucesb/database checks ok

06:00 AIDA crashes, lose connection to aida09

06:20 aida08 does not respond on powercycle and reset

06:30 Called TD 

06:50 successfully reset AIDA. Writing to file R30

      synchronise ASIC clocks:
	
FEE64 module aida01  => 	7
FEE64 module aida02  => 	7
FEE64 module aida03  => 	7
FEE64 module aida04  => 	7
FEE64 module aida05  => 	7
FEE64 module aida06  => 	7
FEE64 module aida07  => 	7
FEE64 module aida08  => 	7
FEE64 module aida09  => 	7
FEE64 module aida10  => 	7
FEE64 module aida11  => 	7
FEE64 module aida12  => 	7


ASIC Clock Timestamp check test result: Passed 12, Failed 0

Database options file size check:


If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

System wide checks ok *except*

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

re-calibration of these FEE modules unsuccessful in FADC align and control.

07:05 FEE64 Temps ok - attachment 12

      good event statistics ok - attachment 13

      detector bias / leakage currents ok - attachment 14

07:10 stats/ucesb/database checks ok

07:30 stats/ucesb/database checks ok


      
      
Attachment 1: Screenshot_from_2021-04-16_23-35-20.png
Screenshot_from_2021-04-16_23-35-20.png
Attachment 2: Screenshot_from_2021-04-16_23-52-09.png
Screenshot_from_2021-04-16_23-52-09.png
Attachment 3: Screenshot_from_2021-04-17_00-05-06.png
Screenshot_from_2021-04-17_00-05-06.png
Attachment 4: Screenshot_from_2021-04-17_00-31-46.png
Screenshot_from_2021-04-17_00-31-46.png
Attachment 5: Screenshot_from_2021-04-17_01-11-18.png
Screenshot_from_2021-04-17_01-11-18.png
Attachment 6: Screenshot_from_2021-04-17_01-12-26.png
Screenshot_from_2021-04-17_01-12-26.png
Attachment 7: Screenshot_from_2021-04-17_01-13-22.png
Screenshot_from_2021-04-17_01-13-22.png
Attachment 8: 20210417_0506.png
20210417_0506.png
Attachment 9: 20210417_0602_tem.png
20210417_0602_tem.png
Attachment 10: 20210417_0601.png
20210417_0601.png
Attachment 11: 20210417_0603_Bias.png
20210417_0603_Bias.png
Attachment 12: Screenshot_from_2021-04-17_06-03-52.png
Screenshot_from_2021-04-17_06-03-52.png
Attachment 13: Screenshot_from_2021-04-17_06-04-14.png
Screenshot_from_2021-04-17_06-04-14.png
Attachment 14: Screenshot_from_2021-04-17_06-04-37.png
Screenshot_from_2021-04-17_06-04-37.png
  231   Fri Apr 16 15:52:08 2021 DSJ 16 April 16.00 shift
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23     10      3      1      1      2      1      3      3      3      6   : 36156
aida02 :     25     10      4      3      1      2      2      3      3      3      6   : 36500
aida03 :     27      8      5      2      3      4      3      3      2      3      6   : 36092
aida04 :     30      4      4      5      0      2      4      2      3      3      6   : 36472
aida05 :     19      5      7      1      3      2      2      2      2      4      6   : 37060
aida06 :     18     13      1      2      1      3      2      3      3      3      6   : 36544
aida07 :     24      7      3      2      2      3      1      3      2      4      6   : 37384
aida08 :     15     11      2      5      2      4      1      4      3      3      6   : 37076
aida09 :      7      3      4      1      5      6      5      4      3      2      6   : 36308
aida10 :      8      3     13      4      3      3      1      4      2      3      6   : 36040
aida11 :     18     11     17      7      4      4      4      3      2      3      6   : 36752
aida12 :     12      7      7      7      4      4      2      3      2      3      6   : 36024



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021



17:27.  Checked system stats, temps, leakage etc. looks ok 


18:49. Tape server writing to disk /NULL/R21_9+
       /TapeData points to /media/SecondDrive/ - 3.6Tb free

19.15. checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16      9     16      2      2      2      1      3      3      3      6   : 36424
aida02 :      1      3      5      4      1      3      3      2      3      3      6   : 36268
aida03 :     24     13     11      6      3      3      2      4      2      3      6   : 36472
aida04 :      5      4      4      4      3      3      3      2      3      3      6   : 36404
aida05 :     17      8      4      2      2      2      2      2      2      4      6   : 36996
aida06 :     26     11      3      2      0      3      2      3      3      3      6   : 36528
aida07 :     24      7      2      1      1      1      2      3      2      4      6   : 37272
aida08 :     13     12      6      3      3      4      1      4      3      3      6   : 37140
aida09 :     22     15     19      2      6      5      4      4      3      2      6   : 36416
aida10 :     16     11      6      4      3      3      1      4      2      3      6   : 36024
aida11 :     12      4      5      4      2      3      3      3      2      3      6   : 35872
aida12 :     18     12      9      4      4      4      2      3      2      3      6   : 36024


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


19.53.  Checked system stats, temps, leakage etc. looks ok 

20:30.  Checked system stats, temps, leakage etc. looks ok 

20:59.  Checked system stats, temps, leakage etc. looks ok 



21:31 System checks
	
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     21      9     17      3      2      2      1      3      3      3      6   : 36492
aida02 :      5      1      1      6      0      2      2      3      3      3      6   : 36332
aida03 :      2     10     11      6      4      4      3      3      2      3      6   : 36296
aida04 :     27     12      3      4      0      2      3      2      3      3      6   : 36220
aida05 :     19      9      7      2      1      2      2      2      2      4      6   : 36996
aida06 :     28      5      0      4      0      3      1      3      3      3      6   : 36248
aida07 :     19     13     10      6      2      2      1      3      2      4      6   : 37524
aida08 :     18      9      6      0      1      4      1      3      3      3      6   : 36400
aida09 :     17     19     12      3      6      5      4      4      3      2      6   : 36348
aida10 :     23      9      7      4      2      3      1      4      2      3      6   : 35988
aida11 :     19     10      3      3      2      3      3      3      2      3      6   : 35884
aida12 :     12      6      0      3      2      4      2      3      2      3      6   : 35648


	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021


22.06.  Checked system stats, temps, leakage etc. looks ok 

22.18CEST BNC PB-5 amplitude changed from 1V to 2V
          fioler NULL/R21_67

22.23CEST All histograms zero'd

22.43.  Checked system stats, temps, leakage etc. looks ok 
23.21.  Checked system stats, temps, leakage etc. looks ok 


23.46 system checks
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb40 : 	 6  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     17     12      7      0      3      3      2      4      2      3      6   : 36180
aida02 :     35      3      5      2      0      2      1      3      3      3      6   : 36148
aida03 :     22      6     10      2      1      3      2      4      2      3      6   : 36136
aida04 :     27      9      3      5      2      4      3      3      2      3      6   : 36100
aida05 :     20      9      3      3      2      2      2      2      2      4      6   : 37032
aida06 :     25     11      1      2      0      3      1      3      3      3      6   : 36236
aida07 :     20      6      6      2      3      3      2      4      1      4      6   : 37216
aida08 :     25     10      0      2      1      3      1      3      3      3      6   : 36276
aida09 :     10      4      3      4      4      5      4      4      3      2      6   : 35960
aida10 :     26      9      3      0      1      3      1      4      2      3      6   : 35744
aida11 :      2      3      4      2      2      3      3      3      2      3      6   : 35744
aida12 :      4      9      7      2      2      4      2      3      2      3      6   : 35720
	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Fri Apr 16 01:00:12 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021
Attachment 1: Screenshot_2021-04-16_Statistics_aidas-gsi.png
Screenshot_2021-04-16_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 10.png
10.png
Attachment 4: Screenshot_2021-04-16_Statistics_aidas-gsi(1).png
Screenshot_2021-04-16_Statistics_aidas-gsi(1).png
Attachment 5: 11.png
11.png
Attachment 6: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 7: Screenshot_2021-04-16_Statistics_aidas-gsi(2).png
Screenshot_2021-04-16_Statistics_aidas-gsi(2).png
Attachment 8: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 9: 12.png
12.png
Attachment 10: Screenshot_2021-04-16_Statistics_aidas-gsi(3).png
Screenshot_2021-04-16_Statistics_aidas-gsi(3).png
Attachment 11: Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-16_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 12: 13.png
13.png
  230   Fri Apr 16 11:42:45 2021 OHShifter checklist
Every 30 minutes:

- Reload statistics (Web browser tab screen 1, workspace 2)
    - Are all FEEs still showing a rate?
    - If one shows 0 after multiple reloads Call Oscar or Tom.

- Options database monitor (Top middle terminal screen 2, workspace 2)
    - Has the message changed since the last check?
    - If so copy the message and place in elog

- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 4)
    - If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
    - Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 2)
        - Ensure "Act on all FEE" and "Act on all asic" selected
        - Click on drop down menu in the bottom of the screen and select check ASIC control
        - It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
        - A pop up will appear check that all options say ok
        - Has the implant rate gone to a more normal amount?
            - If not feel free to call

Every two hours:

- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 2)
     - Compare to the previous screenshot to check if anything has drastically changed

- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 2)
     - Do they all appear stable?

- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 2)

- Perform the five system wide checks (Web browser tab screen 1, workspace 2)
     - Make a note in the elog of which tests are successful
     - If failed copy the text from the white box into the elog and make a note of it

- Enter the screenshots into the elog

- Update the leakage current spreadsheet (Web browser, screen 2, workspace 4)
  229   Fri Apr 16 07:19:00 2021 OH, MAFriday 16th April 08:00-16:00
08:19 System wide checks all ok
      Baselines reset on WR and FPGA checks for the start of the day

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3

09:03 During the reset last night I repaired the OptionsDB for aida02 from the Options.Pristine directory.
      No corruptions have been noted since

10:10 System check, clcock and ADC ok

	 Base 		Current 	Difference
aida07 fault 	 0xfb3a : 	 0xfb3d : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA and memory check ok

      Statistics ok - attachment 4
      Temperatures ok - attachment 5
      Bias and leakage currents ok - attachment 6

10:39 Fission fragments were punching through AIDA
      This was even with the maxmimum degraders in place
      They are now placing a thick plate in front of AIDA to block the fragments while they beam tune.

11:38 They have found the issue with the degrader and have corrected it.
      We now see next to no implants in AIDA as expect.
      Histograms reset at around 11:35
      Layout 2 shows very few implants in 5 minutes - attachment 7

12:08 The high implantation rates of ions into AIDA is clearly seen in the large transients on the leakage currents - attachment 8
      This was a large amount of dose going into the detectors which we should try to avoid

13:25 system check, clock and ADC ok

	 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics ok - attachment 9
      Temperatures ok - attachment 10
      Bias and leakage currents ok - attachment 11
FPGA and memory check ok


16:10 system check, clock and ADC ok

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


      Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14
FPGA and memory check ok
Attachment 1: 210416_0820_Stats.png
210416_0820_Stats.png
Attachment 2: 210416_0821_Temp.png
210416_0821_Temp.png
Attachment 3: 210416_0824_Bias.png
210416_0824_Bias.png
Attachment 4: 210416_1003Stats.png
210416_1003Stats.png
Attachment 5: 210416_1007Temp.png
210416_1007Temp.png
Attachment 6: 210416_1019_voltages.png
210416_1019_voltages.png
Attachment 7: 210416_1139_layout2.png
210416_1139_layout2.png
Attachment 8: 210416_1207_Leakage.png
210416_1207_Leakage.png
Attachment 9: 210416_1311_Stats.png
210416_1311_Stats.png
Attachment 10: 210416_1312_Temp.png
210416_1312_Temp.png
Attachment 11: 210416_1315_Voltages.png
210416_1315_Voltages.png
Attachment 12: 210416_1412_Voltages.png
210416_1412_Voltages.png
Attachment 13: 210416_1548_Stats.png
210416_1548_Stats.png
Attachment 14: 210416_1553_Temp.png
210416_1553_Temp.png
  228   Fri Apr 16 05:13:09 2021 MLsystem wide checks

system wide checks:

Done around 4am.

#################################

To be noted:

-1- White rabbit and FPGA checks failed for 1 FEE and passed for 11

-2- No waveform for AIDA09 in 1.8.L spectrum

-3- AIDA 7 is red in the NewMerger control window (?)

#################################

 

Details:

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

White Rabbit checks:

         Base         Current     Difference
aida07 fault      0xfa8c :      0xfb37 :      171  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA timestamp check:


             Base         Current         Difference
aida07 fault      0x0 :      0xe :      14  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

Memory check:

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     45      9      7      3      4      3      2      3      3      3      6   : 36940
aida02 :     23      9      6      7      4      3      2      3      3      3      6   : 36964
aida03 :     40      8      5      4      5      3      2      3      3      3      6   : 36976
aida04 :     37     13      7      3      4      3      4      2      3      3      6   : 36940
aida05 :     35     11      6      3      4      4      2      3      2      4      6   : 38052
aida06 :     42      8      2      3      4      4      2      3      3      3      6   : 36968
aida07 :     42     10      5      5      5      3      2      3      3      3      6   : 37032
aida08 :     40      8      1      6      3      4      2      3      3      3      6   : 36976
aida09 :     18     18     14      8      5      5      5      4      3      2      6   : 36728
aida10 :      1      8      9      6      3      3      1      4      2      3      6   : 36052
aida11 :     24     11     14      6      2      2      4      2      3      3      6   : 36824
aida12 :     17     13      8      5      5      4      2      3      2      3      6   : 36108

Attachment 1: 16-04-2021_3h45am_HV.png
16-04-2021_3h45am_HV.png
Attachment 2: 48_temperature.png
48_temperature.png
Attachment 3: 16-04-2021_3h51_StatGoodEvt.png
16-04-2021_3h51_StatGoodEvt.png
Attachment 4: 16-04-2021_3h53_StatADCdataItem.png
16-04-2021_3h53_StatADCdataItem.png
Attachment 5: 16-04-2021_3h58_StatWR#4.png
16-04-2021_3h58_StatWR#4.png
Attachment 6: 16-04-2021_4h04_StatWR#5.png
16-04-2021_4h04_StatWR#5.png
Attachment 7: 16-04-2021_3h59_StatPause#2.png
16-04-2021_3h59_StatPause#2.png
Attachment 8: 16-04-2021_4h01_StatResume#3.png
16-04-2021_4h01_StatResume#3.png
Attachment 9: 16-04-2021_4h02_StatCorrelation#8.png
16-04-2021_4h02_StatCorrelation#8.png
Attachment 10: 16-04-2021_4h06_SpecRate.png
16-04-2021_4h06_SpecRate.png
Attachment 11: 16-04-2021_4h08_SpecRate2.png
16-04-2021_4h08_SpecRate2.png
Attachment 12: 16-04-2021_4h17_Spec1.8L.png
16-04-2021_4h17_Spec1.8L.png
Attachment 13: 16-04-2021_4h29_Spec1.8L_id4.png
16-04-2021_4h29_Spec1.8L_id4.png
Attachment 14: 16-04-2021_4h20_Spec1.8W_id7.png
16-04-2021_4h20_Spec1.8W_id7.png
Attachment 15: 25_Spec1.8W_id8.png
25_Spec1.8W_id8.png
Attachment 16: 16-04-2021_4h48_NewMergerControl.png
16-04-2021_4h48_NewMergerControl.png
  227   Fri Apr 16 02:15:12 2021 NHImplants/Punchthrough
Rate with beam < 1000 Hz still

Not Uranium but some fragments possibly produced in the thick 9g degrader
S4 slits are +/- 20mm but these punchthrough and cover the +/- 40mm of AIDA
Attachment 1: 53_AM.png
53_AM.png
  225   Fri Apr 16 00:47:36 2021 OH-MLAIDA DAQ Reset

1:18 (CET) No rate in AIDA 07.

Oscar did a AIDA DAQ reset and a quick wide check.

(Power cycle

All back to normal.

  224   Thu Apr 15 22:38:18 2021 OHThursday 15th April - 16th April
23:38 They are testing the finger detector
      It is a lot of U they are putting into it which is passing through to AIDA ~3500Hz
      Most seems to be punching through

23:45 Rate seems to be reduced to peaking at 1000Hz

00:06 DAQ Running to no storage
      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa


      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse
  223   Thu Apr 15 13:28:07 2021 TDThursday 15 April
13.28 System wide checks

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida01 fault 	 0x44c2 : 	 0x44c5 : 	 3  
aida02 fault 	 0xc81f : 	 0xc822 : 	 3  
aida03 fault 	 0x9342 : 	 0x9344 : 	 2  
aida04 fault 	 0xdb3f : 	 0xdb41 : 	 2  
aida05 fault 	 0x4baf : 	 0x4bb3 : 	 4  
aida06 fault 	 0xd2a4 : 	 0xd2a8 : 	 4  
aida07 fault 	 0xc51c : 	 0xc51f : 	 3  
aida08 fault 	 0xe853 : 	 0xe856 : 	 3  
aida09 fault 	 0x5212 : 	 0x5216 : 	 4  
aida10 fault 	 0xf81d : 	 0xf81f : 	 2  
aida11 fault 	 0x810 : 	 0x812 : 	 2  
aida12 fault 	 0xa279 : 	 0xa27a : 	 1  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x5 : 	 5  
aida12 fault 	 0x0 : 	 0xc : 	 12  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     57     33     12     15      3      3      3      2      3      2      6   : 35276
aida02 :     15      3      5      2      1      4      1      3      3      3      6   : 36388
aida03 :     13      4      2      1      4      3      1      3      3      3      6   : 36372
aida04 :     21      5      2      1      2      4      4      2      3      3      6   : 36668
aida05 :     10      5      6      3      2      3      3      3      1      4      6   : 36880
aida06 :     22     12      1      0      3      3      1      3      3      3      6   : 36360
aida07 :     19      3      3      0      3      3      2      4      1      4      6   : 37076
aida08 :     21      6      2      4      2      3      1      3      3      3      6   : 36388
aida09 :      2      7      5      3      2      4      2      3      2      3      6   : 35696
aida10 :      7      3      1      3      1      3      3      3      2      3      6   : 35684
aida11 :      1      2      1      3      2      5      2      2      3      3      6   : 36228
aida12 :     17     10      4      2      3      2      2      4      2      3      6   : 36052



 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida02 =>   Options file size is 2114 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

13.30 Output of ~/oh/OptionsCheck.py

02:25:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&ASIC.se
ttings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

13.37 Detector bias & leakage currents OK - attachment 1
      FEE64 temps OK - attachments 2
      Statistics - good events, ADC data, disc data, info code 4 & 5, pause & resume, correlation - attachments 3-10
      per FEE64 rate spectra - attachments 11 & 12
      per FEE64 1.8.L spectra - attachments 13 & 19
       aida5 peak width 70 ch FWHM, aida06 peak width 89ch FWHM
      per FEE64 1.8.W spectra 20us FSR - attachments 14 & 15
      meger & tape server - attachments 16-18
 
Attachment 1: 4.png
4.png
Attachment 2: Screenshot_2021-04-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_2021-04-15_Statistics_aidas-gsi.png
Screenshot_2021-04-15_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-04-15_Statistics_aidas-gsi(1).png
Screenshot_2021-04-15_Statistics_aidas-gsi(1).png
Attachment 5: Screenshot_2021-04-15_Statistics_aidas-gsi(2).png
Screenshot_2021-04-15_Statistics_aidas-gsi(2).png
Attachment 6: Screenshot_2021-04-15_Statistics_aidas-gsi(3).png
Screenshot_2021-04-15_Statistics_aidas-gsi(3).png
Attachment 7: Screenshot_2021-04-15_Statistics_aidas-gsi(4).png
Screenshot_2021-04-15_Statistics_aidas-gsi(4).png
Attachment 8: Screenshot_2021-04-15_Statistics_aidas-gsi(5).png
Screenshot_2021-04-15_Statistics_aidas-gsi(5).png
Attachment 9: Screenshot_2021-04-15_Statistics_aidas-gsi(6).png
Screenshot_2021-04-15_Statistics_aidas-gsi(6).png
Attachment 10: Screenshot_2021-04-15_Statistics_aidas-gsi(7).png
Screenshot_2021-04-15_Statistics_aidas-gsi(7).png
Attachment 11: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(1).png
Attachment 13: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(2).png
Attachment 14: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(3).png
Attachment 15: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(4).png
Attachment 16: Screenshot_2021-04-15_NewMerger_Control_aidas-gsi.png
Screenshot_2021-04-15_NewMerger_Control_aidas-gsi.png
Attachment 17: Screenshot_2021-04-15_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2021-04-15_Tape_Service_(Expert)_aidas-gsi.png
Attachment 18: 5.png
5.png
Attachment 19: Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-15_Spectrum_Browser_aidas-gsi(5).png
  222   Thu Apr 15 12:23:22 2021 NHFast Discriminator Masks
It is possible to get good rates with the fast discriminators with a few hot channels mask in aida03 and aida10
Threshold is 0x20 = 3.2 MeV

They can be tested in beam to see performance, especially note the first setting will involve alpha emitters to check in AIDA

All FEEs:
AS01 01, AS3 07 10 13 are masked

aida03: 
AS0 12 and 14 are masked in addition

aida10:
AS1 10, 11 and AS2 05, 06 are masked in addiction
Attachment 1: Image_Pasted_at_2021-4-14_18-57.png
Image_Pasted_at_2021-4-14_18-57.png
  221   Wed Apr 14 15:14:55 2021 PJCSNote Well

Please note that for the forseeable future the system function in the menu on the System Wide Checks browser page labelled as 

"Collect the Timestamp RAM values"  and "Start the Readout Timestamp error tracing" 

Are for the use of engineers only unless otherwise instructed. 

There is no harm in using them but the results may be confusing and it is not possible, at present, to detail a definitive set of instructions for their use.

  220   Wed Apr 14 12:14:47 2021 PJCSCorrections and changes

14/4/21 @12:00 UK time

Corrected the failure of FADC re-calibration All modules. The problem was a missing > in the .tml file

Edited the sys.tcl file in /MIDAS/TclHttpd/Html/RunControl to comment out where the status of the waveform enable was able to stop the creation of the .W spectra. This will now function at SETUP.

Edited the sys.tcl file to remove the test for Master module at the enable of the Correlation Scalar readout. All modules will now be enabled.

Successfully re-calibrated the LMK3200 in aida09 to remove the clock fault reported in System Wide Checks.

Successfully "sync ASIC clocks" followed by calibrate all module FADCs ( except aida09 which required seperate attention. )

All the Edits first carried out and tested at Daresbury T9 system.

  219   Wed Apr 14 09:55:12 2021 TDTuesday 13 April
Two further comments

1) The 'pristine' Options files produced by OH fixed *all* of the undefined spectra issues,
   i.e. we now have *L, *H, Rate, Stat, Hit, HitRate spectra for all FEE64s

2) The system has undegone 1x power cycle/reset/setup and 1x reset/setup cycle since the
   new Options files were installed and we currently have 2x errors reported in aida02.
   There is an open Options tab and we have viewed the Options settings for aida01 and aida02
   (currently viewing aida02) - *no* changes/saves/restores requested.

> 
> Rename Options directory
> 
> mv Options Options.BAK-130421
> 
> Copy Options.PRISTINE to Options
> 
> cp -r Options.PRISTINE Options
> chmod -R 777 Options
> 
> Start Options directory monitor (will check directory every 5 minutes)
> 
> Python ~/oh/OptionsCheck
> 
> 
> 17.32 Power cycle FEE64s
> 
> 
> FEE64 module aida09 global clocks failed, 6
>  Clock status test result: Passed 11, Failed 1
> 
> Understand status as follows
> Status bit 3 : firmware PLL that creates clocks from external clock not locked
> Status bit 2 : always logic '1' 
> Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
> Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
> If all these bits are not set then the operation of the firmware is unreliable
> 
> Calibration test result: Passed 12, Failed 0
> 
> If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
> 
> White Rabbit error counter test result: Passed 12, Failed 0
> 
> Understand the status reports as follows:-
> Status bit 3 : White Rabbit decoder detected an error in the received data
> Status bit 2 : Firmware registered WR error, no reload of Timestamp
> Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
> 
> 	
> FPGA Timestamp error counter test result: Passed 12, Failed 0
> If any of these counts are reported as in error
> The ASIC readout system has detected a timeslip.
> That is the timestamp read from the time FIFO is not younger than the last
> 
> 
> 
> Returned 0 0 0 0 0 0 0 0 0 0 0 0  
> Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
> aida01 :      3      5      2      2      1      3      3      2      2      3      8   : 43348
> aida02 :      2      3      0      2      1      2      2      3      2      3      8   : 43424
> aida03 :      3      4      0      1      1      2      2      3      2      3      8   : 43404
> aida04 :      1      4      2      0      3      3      3      2      2      3      8   : 43396
> aida05 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
> aida06 :      2      3      2      1      1      2      2      3      2      3      8   : 43424
> aida07 :      4      1      3      3      1      3      1      3      2      3      8   : 43368
> aida08 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
> aida09 :      2      3      3      3      2      4      2      2      2      3      8   : 43312
> aida10 :      1      3      1      2      2      3      2      2      2      3      8   : 43116
> aida11 :      1      3      3      1      2      3      2      2      2      3      8   : 43116
> aida12 :      3      2      1      0      1      2      1      3      2      3      8   : 43116
> 
> 
> Note that system wide checks identify 1x file of different size immediately after boot/reset/setup
> 
>  Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
> 
>  FEE : aida01 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida02 =>   Options file size is 1025 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida03 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida04 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida05 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida06 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida07 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida08 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida09 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida10 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
>  FEE : aida11 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
>  FEE : aida12 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:05 CEST 2021
> 
> 06:25:00 PM
> FAULTS!
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAV
E_DMA_HWM&&Ai
> da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
> 
> 06:30:00 PM
> FAULTS!
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAV
E_DMA_HWM&&Ai
> da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
> 
> Unknown option in FEE2 Options database
> Unknown entry starts: 
> 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.setting
s&&WAVE_DMA_H
> 
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_En
able
> 
> 
> 
> 
> 
> FEE64 module aida01  => 	7
> FEE64 module aida02  => 	7
> FEE64 module aida03  => 	7
> FEE64 module aida04  => 	7
> FEE64 module aida05  => 	7
> FEE64 module aida06  => 	7
> FEE64 module aida07  => 	7
> FEE64 module aida08  => 	7
> FEE64 module aida09  => 	7
> FEE64 module aida10  => 	7
> FEE64 module aida11  => 	7
> FEE64 module aida12  => 	7
> 
> 
> ASIC Clock Timestamp check test result: Passed 12, Failed 0
> 
> 
>  If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status
> 
> 
> 
> 
>  aida01 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida02 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida03 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida04 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida05 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida06 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida07 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida08 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida09 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida10 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida11 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
>  aida12 timestamp memory	No stop found : 0x0
> 0		0	0	0	0
> 4		0	0	0	0
> 8		0	0	0	0
> 12		0	0	0	0
> 16		0	0	0	0
> 20		0	0	0	0
> 24		0	0	0	0
> 28		0	0	0	0
> 
> 17.14 rate spectra - attachment 1
> 
>       1.8.L spectra - attachments 2 & 3
>       pulser peak width aida06 c.75 ch FWHM, aida07  c.85 ch FWHM
> 
>       1.8.W spectra - attachments 4 & 5
> 
>       good events statistics (slow comparator 0xa) - attachment 6
> 
>       FEE64 temperatures OK - attachment 7
>     
>       DSSSD bias & leakage currents OK - attachment 8
> 
> 
> 
> 
> 17.38 DAQ starts file NULL/R14
>       slow comparator 0xa -> 0x64
>       BNC PB-5 pulser OFF
>       alpha background
> 
> 10:40 CET 14.04.2021 - NH
>       Close file NULL/R14
  218   Tue Apr 13 16:49:41 2021 TDTuesday 13 April
Rename Options directory

mv Options Options.BAK-130421

Copy Options.PRISTINE to Options

cp -r Options.PRISTINE Options
chmod -R 777 Options

Start Options directory monitor (will check directory every 5 minutes)

Python ~/oh/OptionsCheck


17.32 Power cycle FEE64s


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

Calibration test result: Passed 12, Failed 0

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last



Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      3      5      2      2      1      3      3      2      2      3      8   : 43348
aida02 :      2      3      0      2      1      2      2      3      2      3      8   : 43424
aida03 :      3      4      0      1      1      2      2      3      2      3      8   : 43404
aida04 :      1      4      2      0      3      3      3      2      2      3      8   : 43396
aida05 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
aida06 :      2      3      2      1      1      2      2      3      2      3      8   : 43424
aida07 :      4      1      3      3      1      3      1      3      2      3      8   : 43368
aida08 :      1      4      2      2      2      3      3      2      2      3      8   : 43396
aida09 :      2      3      3      3      2      4      2      2      2      3      8   : 43312
aida10 :      1      3      1      2      2      3      2      2      2      3      8   : 43116
aida11 :      1      3      3      1      2      3      2      2      2      3      8   : 43116
aida12 :      3      2      1      0      1      2      1      3      2      3      8   : 43116


Note that system wide checks identify 1x file of different size immediately after boot/reset/setup

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:04 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:03 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Tue Apr 13 17:59:05 CEST 2021

06:25:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

06:30:00 PM
FAULTS!

Unknown option in FEE2 Options database
Unknown entry starts: 
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Ai
da_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable

Unknown option in FEE2 Options database
Unknown entry starts: 
et&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_H
WM&&Aida_Hist_D_Enable&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable





FEE64 module aida01  => 	7
FEE64 module aida02  => 	7
FEE64 module aida03  => 	7
FEE64 module aida04  => 	7
FEE64 module aida05  => 	7
FEE64 module aida06  => 	7
FEE64 module aida07  => 	7
FEE64 module aida08  => 	7
FEE64 module aida09  => 	7
FEE64 module aida10  => 	7
FEE64 module aida11  => 	7
FEE64 module aida12  => 	7


ASIC Clock Timestamp check test result: Passed 12, Failed 0


 If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status




 aida01 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida02 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida03 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida04 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida05 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida06 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida07 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida08 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida09 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida10 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida11 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

 aida12 timestamp memory	No stop found : 0x0
0		0	0	0	0
4		0	0	0	0
8		0	0	0	0
12		0	0	0	0
16		0	0	0	0
20		0	0	0	0
24		0	0	0	0
28		0	0	0	0

17.14 rate spectra - attachment 1

      1.8.L spectra - attachments 2 & 3
      pulser peak width aida06 c.75 ch FWHM, aida07  c.85 ch FWHM

      1.8.W spectra - attachments 4 & 5

      good events statistics (slow comparator 0xa) - attachment 6

      FEE64 temperatures OK - attachment 7
    
      DSSSD bias & leakage currents OK - attachment 8




17.38 DAQ starts file NULL/R14
      slow comparator 0xa -> 0x64
      BNC PB-5 pulser OFF
      alpha background

10:40 CET 14.04.2021 - NH
      Close file NULL/R14
Attachment 1: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(2).png
Attachment 4: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(3).png
Attachment 5: Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-13_Spectrum_Browser_aidas-gsi(4).png
Attachment 6: Screenshot_2021-04-13_Statistics_aidas-gsi.png
Screenshot_2021-04-13_Statistics_aidas-gsi.png
Attachment 7: Screenshot_2021-04-13_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-13_Temperature_and_status_scan_aidas-gsi.png
Attachment 8: 3.png
3.png
  217   Tue Apr 13 14:24:48 2021 TDS460 information
aida-gsi Anydesk 897170655

DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/

S460 shift plan - https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1097596683

Mattermost - https://mattermost.gsi.de/despec/channels/aida

DESPEC Zoom meeting

Zoom Meeting https://gsi-fair.zoom.us/j/94404952361
Passcode: s460
  216   Mon Apr 12 18:36:31 2021 NH[HowTo] AIDA & MBS
Some instructions on AIDA and MBS and troubleshooting are available here
https://sf.gsi.de/lib/c9ba0b4c-26bc-43d0-9b49-213fc594610f/file/AIDA%20and%20MBS%20Tips%20and%20Troubleshooting.pdf
Attachment 1: AIDA_and_MBS_Tips_and_Troubleshooting.pdf
AIDA_and_MBS_Tips_and_Troubleshooting.pdf AIDA_and_MBS_Tips_and_Troubleshooting.pdf AIDA_and_MBS_Tips_and_Troubleshooting.pdf
  215   Mon Apr 12 16:49:40 2021 TDMonday 22 April - pulser walkthrough & alpha background
16.54 Pulser walkthrough

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      file NULL/R4

      BNC PB-5 settings changed by terminal command line using the commands

      set rep rate 25
      set attenuation 5
      set amplitude 10
      set amplitude 9
      :
      :
      set amplitude 1
      set attenuation 1
      set rep rate 2

      aida07 & aida08 1.8.L spectra - see attachment 1

17.26 run ends OK file NULL/R4_27

17.26 file NULL/R5

      amplitudes 0.6, 1, 1.4, 1.8, 2.2, 2.6, 3

      aida07 & aida08 1.8.L spectra - see attachment 2

17.47 run ends OK NULL/R5_23

17.53 system wide checks

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida09 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     16     11      3      2      0      2      4      2      3      4     11   : 58888
aida02 :      7     11      4      2      0      4      3      2      3      3      7   : 40436
aida03 :     38      7      7      2      6      4      1      3      3      3      6   : 36864
aida04 :     19     10      5      2      1      3      1      4      2      4     15   : 74732
aida05 :     30      5      4      3      2      3      4      3      3      4     15   : 76096
aida06 :     24      4      3      3      1      4      2      3      2      4      7   : 41808
aida07 :     22      7     11      7      1      6      3      4      3      3      6   : 37984
aida08 :     13      7      5      2      1      4      3      3      1      4      6   : 36924
aida09 :     33      9     10      6      4      4      2      2      2      4      6   : 37676
aida10 :     15      7      7      1      1      3      1      3      2      3      7   : 39364
aida11 :     30     13      8      3      4      4      3      3      3      3      6   : 37312
aida12 :     17      9      4      3      2      3      2      2      3      3      7   : 40236

17.57 Slow comparator 0xa -> 0x64
      BNC PB-5 Pulser OFF

      file NULL/R9
      alpha background

      FEE64 temperatures OK - see attachment 3
      statistics OK - see attachment 4
      DSSSD bias & leakage currents OK - see attachment 5
      tape server & merger - attachment 6

11:31 13.04.21 
      file NULL/R9 stopped for work in day
      will resume alpha over night again

 
Attachment 1: Screenshot_2021-04-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-12_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-04-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-04-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 4: Screenshot_2021-04-12_Statistics_aidas-gsi.png
Screenshot_2021-04-12_Statistics_aidas-gsi.png
Attachment 5: 1.png
1.png
Attachment 6: 2.png
2.png
  214   Sat Apr 10 12:15:54 2021 OH, TDSaturday 10 April
12:37 DAQ found crashed. Had crashed at some time following 7:50am following previous statistics update
      Unable to recover the FEEs so forced to power cycle
      Following power cycle statistics much improved from yesterday.
      In the waveforms the lower frequency component that was prevalent yesterday is no longer visible.
      The waveforms are not perfect by any means though
      Also note that following an ASIC synchronisation all of the FEEs lose ADC calibration and must be manually recalibrated.

16.42 UTC+1

      per FEE64 1.8.W spectra

      20us FSR attachments 7 & 8
      200us FSR attachments 9 & 10
      2ms FSR attachments 11 & 12
      20ms FSR attachments 15 & 16


      rate spectra attachment 13
      good events statistics attachment 14
Attachment 1: 210410_1237_Aida_crash.png
210410_1237_Aida_crash.png
Attachment 2: 210410_1242_Crash.png
210410_1242_Crash.png
Attachment 3: 210410_1251_aida06_drop.png
210410_1251_aida06_drop.png
Attachment 4: 210410_1312_Waveform1.png
210410_1312_Waveform1.png
Attachment 5: 210304_1313_Waveform2.png
210304_1313_Waveform2.png
Attachment 6: 210410_1318_Stats.png
210410_1318_Stats.png
Attachment 7: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(5).png
Attachment 9: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 10: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(4).png
Attachment 11: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(2).png
Attachment 12: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(3).png
Attachment 13: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(6).png
Attachment 14: Screenshot_2021-04-10_Statistics_aidas-gsi.png
Screenshot_2021-04-10_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(7).png
Attachment 16: Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-04-10_Spectrum_Browser_aidas-gsi(8).png
  213   Fri Apr 9 16:45:29 2021 NH, HA. GA, GZ, OH, TDAIDA Noise
AIDA snout was reconnected and we saw originally a very good reduction in noise from S452 (fig 1)
Occasional bursts of increased rates were noticed, perhaps related to works in S4.

Tape rate at optimal conditions was around 6 MB/s
We still saw some extrinsic noise in waves, two frequencies observed : 1.5 MHz and 100 kHz or so

Figs 2, 3 are at 50 MS/s
Figs 4, 5 are at 5 MS/s

Today the situation has become a lot worse for reasons still unknown.
aida01 shows very strong noise, frequency 125 kHz or so it seems (still at 5 MS/s)
Amplitude 100 mV or so I think

Investigation still underway
Attachment 1: Image_Pasted_at_2021-4-7_18-15.png.jpeg
Image_Pasted_at_2021-4-7_18-15.png.jpeg
Attachment 2: Image_Pasted_at_2021-4-8_16-142.png.jpeg
Image_Pasted_at_2021-4-8_16-142.png.jpeg
Attachment 3: Image_Pasted_at_2021-4-8_16-14.png.jpeg
Image_Pasted_at_2021-4-8_16-14.png.jpeg
Attachment 4: Image_Pasted_at_2021-4-8_16-17.png.jpeg
Image_Pasted_at_2021-4-8_16-17.png.jpeg
Attachment 5: Image_Pasted_at_2021-4-8_16-18.png.jpeg
Image_Pasted_at_2021-4-8_16-18.png.jpeg
Attachment 6: noise_aida_090421.png
noise_aida_090421.png
  212   Fri Apr 9 16:40:36 2021 NHnnrpi1 Update
nnrpi1 is now back in S4 with a new SD card and new raspberry pi.
The freezing originally continued (it was the USB system failing) but was fixed with a change to /boot/cmdline.txt coherent_pool=4M
It seems the kernel update to fix one bug introduced another... 
Now it is running again with all TTYs connected. I think one TTY may not be working but 11 should be
  211   Wed Mar 31 15:46:09 2021 PJCSHowTo : Synchronize the ASIC clocks.

To Synchronize the FEE64 ASIC clocks to rise at the same timestamp time use the System function Synchronise the ASIC clocks in the System Wide Checks browser page.

The Server will read the current timestamp value and calculate, based on the number of FEEs and the access delay, a timestamp value sometime in the future. F_stamp

F_stamp is written to each FEE and the synchronization is enabled. 

In a state machine in the FEE ASIC clock control, logic is enabled to synchronize its ASIC clock. It compares the current timestamp with F_stamp and if the current timestamp is >= F_stamp then it will start the ASIC clock.

The report in the browser log window gives the 3 LSBs of the timestamp at the instant the ASIC clock is started. They should all be the same. They are from the 10ns timestamp.

 

  210   Wed Mar 31 15:40:42 2021 PJCSCheck Options files function added

There is a new operation available in the System Functions menu in the System Wide Checks page. ( The page needs to be reset once when the FEEs are powered before this function will appear )

Check the Options files are all the same size

Using this function will provide a list of the file sizes of the Options/< fee name >/CONTENTS files along with the last time they were accessed as a list.

It's up to the user, at present, to interpret the results but more is possible if required ....

  209   Wed Mar 31 12:46:10 2021 PJCSHowTo : Calibrate the LMK3200 clock devices

Here is a document showing how and where to calibrate the LMK3200 devices that lock to the system clock and generate the clocks for the ADCs and the FPGA internal logic.

Attachment 1: Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf
Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf Calibrating_the_LMK3200_clock_devices_on_the_AIDA_FEE64.pdf
  208   Tue Mar 30 13:21:50 2021 NHnnrpi1 (tty logs)
We had issues with all TTYs reading out on nnrpi1, I have moved them from an apparently faulty USB hub to a new one.
Trying to reboot the pi to detect the new USB ports and it didn't recover from boot
Trying to repair SD card but I get a lot of errors on my laptop, indications are the SD card is completely dead
A new SD card will be ordered and nnrpi1 restored from the original images

A new USB hub should also be organised as it seems one is faulty (pi hangs if 3 or more USB devices connected to it)
  207   Thu Mar 25 18:04:51 2021 TDMSL type BB18(DS) 24cm x 8cm cabling & bias configuration
Attachment 1: MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf
MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf MSL_type_BB18(DS)_24cm_x_8cm_configuration.pdf
  206   Wed Mar 24 10:31:03 2021 TDAIDA to do (or to understand) list
In no particular order

1) Options

variables unexpectedly changing or becoming unset/undefined - see https://elog.ph.ed.ac.uk/DESPEC/205

Is this why *.L, *.H and Rate spectra are undefined in some FEE64s?


2) Merger 'Bad Timestamp' error reports - e.g. see https://elog.ph.ed.ac.uk/DESPEC/203

Meaning of report? Significance/impact?


3) Large numbers of PAUSE/RESUME for some (but not all) FEE64s - see e.g. https://elog.ph.ed.ac.uk/DESPEC/203

Significance/impact?

-- NH to add

4) Slow but non-zero increase of White Rabbit errors (mostly timestamp but much rarer ASIC errors)
We know the White Rabbit network in general is reliable but could we see interruption to the 200 MHz clock or 10 kHz timestamp when FATIMA DAQ is busy/restarting
Wasn't uniform over all FEEs implying a more local source though?

-- OH to add
5) The dropping of links between FEEs and the MERGER. Quite often these were correlated with the 'Bad Timestamp' error messages and also the WR errors

---- PJCS to add
6) The addition of the NIM clock-divider to allow better ASIC clock synchronisation.
  205   Fri Mar 19 17:16:45 2021 TDFriday 19 March
Screenshots of successive displays of Options for FEE64s aida01, 02, 03 and 04.
Attachment 1: Screenshot_2021-03-19_Options_aidas-gsi.png
Screenshot_2021-03-19_Options_aidas-gsi.png
Attachment 2: Screenshot_2021-03-19_Options_aidas-gsi(4).png
Screenshot_2021-03-19_Options_aidas-gsi(4).png
Attachment 3: Screenshot_2021-03-19_Options_aidas-gsi(1).png
Screenshot_2021-03-19_Options_aidas-gsi(1).png
Attachment 4: Screenshot_2021-03-19_Options_aidas-gsi(5).png
Screenshot_2021-03-19_Options_aidas-gsi(5).png
Attachment 5: Screenshot_2021-03-19_Options_aidas-gsi(2).png
Screenshot_2021-03-19_Options_aidas-gsi(2).png
Attachment 6: Screenshot_2021-03-19_Options_aidas-gsi(6).png
Screenshot_2021-03-19_Options_aidas-gsi(6).png
Attachment 7: Screenshot_2021-03-19_Options_aidas-gsi(3).png
Screenshot_2021-03-19_Options_aidas-gsi(3).png
Attachment 8: Screenshot_2021-03-19_Options_aidas-gsi(7).png
Screenshot_2021-03-19_Options_aidas-gsi(7).png
  204   Thu Mar 18 11:53:10 2021 TDThursday 18 March
12.40 The snout w/bPlast plus 3xDSSSDs was removed yesterday

      AIDA pulser tests via adaptor PCB test inputs, i.e. zero capacitative load and zero leakage current

      per FEE64 1.8.W spectra - attachments 1-4
      per FEE64 Rate spectra - attachment 5
      per FEE64 1.8.L spectra - attachments 6-7 

From the per FEE64 1.8.L

FEE64 FWHM

1  -
2  45
3  25
4  -
5  -
6  45
7  25
8  37
9  30
10 38
11 19
12 35

For comparison we would expect to obtain ~ 15-20 ch FWHM (~10.5-14 keV FWHM) based on tests
conducted at T9, STFC DL. The peak widths observed above are significantly worse than expected.
Are the adaptor PCBs currently grounded to FEE64 Cu mezzanine cover?
Attachment 1: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi.png
Attachment 2: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(1).png
Attachment 3: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(2).png
Attachment 4: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(3).png
Attachment 5: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(4).png
Attachment 6: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(5).png
Attachment 7: Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-03-18_Spectrum_Browser_aidas-gsi(6).png
  203   Wed Mar 17 10:31:09 2021 TDWednesday 17 March
11.28 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

10.26 System wide checks
	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     25     16      3      6      1      3      2      2      3      4     10   : 54676
aida02 :     17     13     14     10      5      5      2      3      3      3      6   : 37516
aida03 :     19     10      8      3      2      3      2      4      2      3      6   : 36220
aida04 :      2      6      2      3      3      2      2      3      2      4     15   : 74360
aida05 :     89     48     20      3      3     34     21     20      6      4      7   : 64324
aida06 :     31     11      1     10      0      1      1      3      4      2      5   : 31140
aida07 :     21     13      8      4      3      2      4      4      3      3      6   : 37756
aida08 :     37     10      2      0      1      2      2      4      3      3      6   : 36932
aida09 :      4     18      6      3      1      3      1      2      3      3      6   : 35872
aida10 :     20      4      1      3      0      2      2      3      3      3      6   : 36320
aida11 :      2      6      6      2      2      4      2      3      2      3      6   : 35672
aida12 :     15     10      2      3      1      2      2      3      3      3      6   : 36428


      FEE64 Temperatures OK - attachment 1
      Good event, disc info #6 & good wave statistics - attachments 2, 3 & 4
           note high disc rates (c. 200k) in FEE64s aida03, aida05 & aida07
      Detector bias & leakage currents OK - attachment 5
      Merger OK - 4.5M data items/s
      TapeServer OK - 15Mb/s

      Rate spectra - attachment 6

11.41 Merger errors since yesterday's Elog


MERGE Data Link (20502): bad timestamp  2 3 0xc08e4cce 0x0c1c8460 0x0000f14c5c1c8460 0x166cf14c5c1c8460 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0967f4f 0x0c1c8460 0x0000f14c5c1c8460 0x166cf14c5c1c8460 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0807dc7 0x0c1c9bd0 0x0000f14c5c1c9bd0 0x166cf14c5c1c9bd0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0957ee9 0x0c1c9bd0 0x0000f14c5c1c9bd0 0x166cf14c5c1c9bd0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0857eea 0x0c1ca3a0 0x0000f14c5c1ca3a0 0x166cf14c5c1ca3a0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0977df3 0x0c1ca3a0 0x0000f14c5c1ca3a0 0x166cf14c5c1ca3a0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc08a7edd 0x0c1cab70 0x0000f14c5c1cab70 0x166cf14c5c1cab70 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7c7f 0x0c1cab70 0x0000f14c5c1cab70 0x166cf14c5c1cab70 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc09d7fd8 0x0c1cb340 0x0000f14c5c1cb340 0x166cf14c5c1cb340 0x166cf1515c1c74c0

 :
 :

MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c338 0x000018ff4433c338 0x166d18ff4433c338 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c356 0x000018ff4433c356 0x166d18ff4433c356 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263f080 0x0433c36a 0x000018ff4433c36a 0x166d18ff4433c36a 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x82630f7f 0x0433c374 0x000018ff4433c374 0x166d18ff4433c374 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c388 0x000018ff4433c388 0x166d18ff4433c388 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3a6 0x000018ff4433c3a6 0x166d18ff4433c3a6 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263efff 0x0433c3c4 0x000018ff4433c3c4 0x166d18ff4433c3c4 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3d8 0x000018ff4433c3d8 0x166d18ff4433c3d8 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3f6 0x000018ff4433c3f6 0x166d18ff4433c3f6 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263efff 0x0433c414 0x000018ff4433c414 0x166d18ff4433c414 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x82218ff4 0x0433c41e 0x000018ff4433c41e 0x166d18ff4433c41e 0x166d18ff4e17c2c0

From Tuesday, March 16, 2021 9:54:36.584 PM to Wednesday, March 17, 2021 10:02:05.935 AM

11.46 Grafana record of bias & leakage current for the 7 days to date - attachment 7

11.51 ASIC check load - all FEE64s, all ASICs

11.51 Disable 'no storage' mode and start writing to disk file S452/R58

12.01 R58_18 & R58_19 *no* change

12.16  R58_24 & R58_25 *all* waveforms disabled (Run Control tab) 

12.23 R58_27 & R58_28 *all* discriminators disabled (Discriminator tab) *but* disc rates now low - see attachment 8 cf. attachment 3

12.27 Enable 'no storage' mode

12.38 Analysis of files R58_19, R58_25 & R58_28 - see attachments 9, 10 & 11
      Disabling waveforms significantly reduces pause/resume for aida01 - aida08
      Effect of disabling discriminators less clear - disc rate zero for all FEE64s except aida03 (2k).
Attachment 1: Screenshot_2021-03-17_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-17_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-17_Statistics_aidas-gsi.png
Screenshot_2021-03-17_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-03-17_Statistics_aidas-gsi(1).png
Screenshot_2021-03-17_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-03-17_Statistics_aidas-gsi(2).png
Screenshot_2021-03-17_Statistics_aidas-gsi(2).png
Attachment 5: 84.png
84.png
Attachment 6: Screenshot_2021-03-17_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-17_Spectrum_Browser_aidas-gsi.png
Attachment 7: 85.png
85.png
Attachment 8: Screenshot_2021-03-17_Statistics_aidas-gsi(3).png
Screenshot_2021-03-17_Statistics_aidas-gsi(3).png
Attachment 9: R58_19
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260603204 ( 1825295.6 Hz)
        Other data format:    1316796 (    9223.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1800 (      12.6 Hz)
                               RESUME:       1800 (      12.6 Hz)
                              SYNC100:        532 (       3.7 Hz)
                              WR48-63:        532 (       3.7 Hz)
                           FEE64 disc:     351517 (    2462.1 Hz)
                             MBS info:     960615 (    6728.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          2 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      142.773 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.872              137.171
  2                2.370              140.392
  3                4.557              140.929
  4               12.507              102.274
  5                9.566              138.244
  6                0.000              130.460
  7                0.043              132.607
  8               21.086              137.171
  9               14.355              142.539
 10                0.094              126.165
 11               38.783              127.507
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   21184711       1567          0          0        302        302         78         78          0        807          0          0
  1   26163497        939          0          0          5          5         40         40          0        849          0          0
  2   26414580     834315          0          0          9          9         78         78     351513     482628          0          0
  3   30144417     476485          0          0         23         23         54         54          0     476331          0          0
  4    6641111       1448          0          0        716        716          8          8          0          0          0          0
  5   18794955        716          0          0        334        334         24         24          0          0          0          0
  6   13042480         36          0          0          0          0         18         18          0          0          0          0
  7    9502247         42          0          0          1          1         20         20          0          0          0          0
  8   31490514        302          0          0        109        109         42         42          0          0          0          0
  9   33799793        352          0          0         86         86         88         88          4          0          0          2
 10   11018662         96          0          0          2          2         46         46          0          0          0          0
 11   32406237        498          0          0        213        213         36         36          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:45075.926s (    0.710 blocks/s,   0.044 Mb/s)
Attachment 10: R58_25
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261566975 ( 1848651.9 Hz)
        Other data format:     353025 (    2495.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        498 (       3.5 Hz)
                               RESUME:        497 (       3.5 Hz)
                              SYNC100:        528 (       3.7 Hz)
                              WR48-63:        528 (       3.7 Hz)
                           FEE64 disc:     350128 (    2474.6 Hz)
                             MBS info:        846 (       6.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      141.491 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.252              140.123
  2                1.261              137.707
  3                1.206              132.607
  4                0.000              122.675
  5                0.045              124.017
  6                0.225              128.312
  7                0.137              137.171
  8               25.852              127.238
  9               22.485              137.976
 10                0.119              139.318
 11               34.852              135.828
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22601243       1010          0          0          5          5         77         77          0        846          0          0
  1   25817640        118          0          0         10         10         49         49          0          0          0          0
  2   26662668     350294          0          0         10         10         73         73     350128          0          0          0
  3   31065481        116          0          0          9          9         49         49          0          0          0          0
  4    7606752         38          0          0          0          0         19         19          0          0          0          0
  5   20370392         52          0          0          3          3         23         23          0          0          0          0
  6   13822142         52          0          0          5          5         21         21          0          0          0          0
  7    9358078         60          0          0          5          5         25         25          0          0          0          0
  8   31862906        361          0          0        132        131         49         49          0          0          0          0
  9   30480452        364          0          0        115        115         67         67          0          0          0          0
 10    7772769         92          0          0          5          5         41         41          0          0          0          0
 11   34146452        468          0          0        199        199         35         35          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:45119.996s (    0.709 blocks/s,   0.044 Mb/s)
Attachment 11: R58_28
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261917114 ( 1867522.0 Hz)
        Other data format:       2886 (      20.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        500 (       3.6 Hz)
                               RESUME:        500 (       3.6 Hz)
                              SYNC100:        523 (       3.7 Hz)
                              WR48-63:        523 (       3.7 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:        840 (       6.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          1 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      140.248 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.196              135.828
  2                1.735              133.949
  3                1.362              133.412
  4                0.000              133.949
  5                0.624              134.218
  6                0.297              110.595
  7                0.085              138.513
  8               26.310              131.265
  9               16.413              138.781
 10                0.037              139.318
 11               35.860              131.265
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22486922       1022          0          0          7          7         84         84          0        840          0          0
  1   25724429         84          0          0          9          9         33         33          0          0          0          0
  2   26397664        186          0          0          7          7         86         86          0          0          0          0
  3   31045386        104          0          0         11         11         41         41          0          0          0          0
  4    7561349         28          0          0          0          0         14         14          0          0          0          0
  5   20305012         70          0          0          8          8         27         27          0          0          0          0
  6   14228124         40          0          0          6          6         14         14          0          0          0          0
  7    9428091         66          0          0          3          3         30         30          0          0          0          0
  8   31422421        379          0          0        144        145         45         45          0          0          0          0
  9   32040015        335          0          0         92         91         76         76          0          0          0          1
 10    7197353         72          0          0          2          2         34         34          0          0          0          0
 11   34080348        500          0          0        211        211         39         39          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:45164.730s (    0.709 blocks/s,   0.044 Mb/s)
  202   Tue Mar 16 09:26:47 2021 TDTuesday 16 March
10.25 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

10.26 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15     19      5      5      1      3      2      2      3      4     10   : 54660
aida02 :     20     13     14     11      5      5      2      4      2      3      6   : 37048
aida03 :      2      3      7      3      2      3      2      4      2      3      6   : 36080
aida04 :     14     10      4      3      3      2      2      4      1      4     15   : 73960
aida05 :     97     51     19      3      3     34     21     20      6      4      7   : 64364
aida06 :     24      7      5      9      0      1      1      3      4      2      5   : 31112
aida07 :     21     13      9      3      3      2      3      4      3      3      6   : 37484
aida08 :     38     10      3      5      2      3      1      4      3      3      6   : 37048
aida09 :     24     11      4      3      2      3      2      3      3      3      6   : 36696
aida10 :     26     10      1      3      1      2      1      4      3      3      6   : 36712
aida11 :      0      7      7      1      1      4      2      3      2      3      6   : 35592
aida12 :      6      8      3      2      2      1      2      3      3      3      6   : 36296


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.5M data items/s
      TapeServer OK - 13Mb/s

      No merger server error messages since most recent check yesterday

15.50 Merger error messages


 :
 :

MERGE Data Link (20503): bad timestamp  3 3 0x838153cb 0x090bf960 0x0000c2c4f90bf960 0x166cc2c4f90bf960 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x090bf960 0x0000c2c4f90bf960 0x166cc2c4f90bf960 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d481de 0x090bff32 0x0000c2c4f90bff32 0x166cc2c4f90bff32 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c181f8 0x090c0ed2 0x0000c2c4f90c0ed2 0x166cc2c4f90c0ed2 0x166cc2c4fefc5fd2
MERGE Actor (20516): Working with 0 from 12 data sources

MERGE Data Link (20503): bad timestamp  3 3 0xc0c47f9e 0x090c16a2 0x0000c2c4f90c16a2 0x166cc2c4f90c16a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d08127 0x090c16a2 0x0000c2c4f90c16a2 0x166cc2c4f90c16a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c681ee 0x090c1e72 0x0000c2c4f90c1e72 0x166cc2c4f90c1e72 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d181c9 0x090c1e72 0x0000c2c4f90c1e72 0x166cc2c4f90c1e72 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98204 0x090c2642 0x0000c2c4f90c2642 0x166cc2c4f90c2642 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc8075 0x090c2e12 0x0000c2c4f90c2e12 0x166cc2c4f90c2e12 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cf812f 0x090c35e2 0x0000c2c4f90c35e2 0x166cc2c4f90c35e2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c780ef 0x090c4d52 0x0000c2c4f90c4d52 0x166cc2c4f90c4d52 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f7816c 0x090c8bd2 0x0000c2c4f90c8bd2 0x166cc2c4f90c8bd2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d480f1 0x090c93a2 0x0000c2c4f90c93a2 0x166cc2c4f90c93a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c48039 0x090ca342 0x0000c2c4f90ca342 0x166cc2c4f90ca342 0x166cc2c4fefc5fd2

 :
 :

MERGE Data Link (20502): bad timestamp  2 3 0xc08d7cc1 0x085c7e10 0x0000c429685c7e10 0x166cc429685c7e10 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08e281c 0x085c85e0 0x0000c429685c85e0 0x166cc429685c85e0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7cc1 0x085c8db0 0x0000c429685c8db0 0x166cc429685c8db0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927e8c 0x085c9580 0x0000c429685c9580 0x166cc429685c9580 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08c414b 0x085ca520 0x0000c429685ca520 0x166cc429685ca520 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857dee 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987d2c 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07d90 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0877fd6 0x085d02e0 0x0000c429685d02e0 0x166cc429685d02e0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927e7c 0x085d2220 0x0000c429685d2220 0x166cc429685d2220 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857d7f 0x085d4930 0x0000c429685d4930 0x166cc429685d4930 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987cc0 0x085d4930 0x0000c429685d4930 0x166cc429685d4930 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07ccd 0x085d5100 0x0000c429685d5100 0x166cc429685d5100 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08e80fd 0x085d7fe0 0x0000c429685d7fe0 0x166cc429685d7fe0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0977e77 0x085d8f80 0x0000c429685d8f80 0x166cc429685d8f80 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0817ec8 0x085d9750 0x0000c429685d9750 0x166cc429685d9750 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0847fca 0x085d9f20 0x0000c429685d9f20 0x166cc429685d9f20 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08c40e8 0x085da6f0 0x0000c429685da6f0 0x166cc429685da6f0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927f3e 0x085db690 0x0000c429685db690 0x166cc429685db690 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0x82614000 0x085dbb36 0x0000c429685dbb36 0x166cc429685dbb36 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857ebc 0x085ddda0 0x0000c429685ddda0 0x166cc429685ddda0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987eec 0x085ddda0 0x0000c429685ddda0 0x166cc429685ddda0 0x166cc429685df380

21.04 Merger errors since 15.50

MERGE Data Link (20502): bad timestamp  2 3 0xc0907de6 0x03087130 0x0000ea6805fd8d80 0x166cea6803087130 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0878007 0x030880d0 0x0000ea68030880d0 0x166cea68030880d0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0937ec9 0x030888a0 0x0000ea68030888a0 0x166cea68030888a0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4e2d 0x0308a7e0 0x0000ea680308a7e0 0x166cea680308a7e0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0998001 0x0308bf50 0x0000ea680308bf50 0x166cea680308bf50 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4abc 0x0308fdd0 0x0000ea680308fdd0 0x166cea680308fdd0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0907fb1 0x030905a0 0x0000ea68030905a0 0x166cea68030905a0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0937fd8 0x03091d10 0x0000ea6803091d10 0x166cea6803091d10 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0807df1 0x03093480 0x0000ea6803093480 0x166cea6803093480 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d14 0x03093c50 0x0000ea6803093c50 0x166cea6803093c50 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x8280dbb0 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x82816a12 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e07 0x030953c0 0x0000ea68030953c0 0x166cea68030953c0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0967c2a 0x030953c0 0x0000ea68030953c0 0x166cea68030953c0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0887d0f 0x03095b90 0x0000ea6803095b90 0x166cea6803095b90 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0997dad 0x03095b90 0x0000ea6803095b90 0x166cea6803095b90 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7bfc 0x03096360 0x0000ea6803096360 0x166cea6803096360 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7c98 0x03096360 0x0000ea6803096360 0x166cea6803096360 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc09d7f7e 0x03096b30 0x0000ea6803096b30 0x166cea6803096b30 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e10 0x03098a70 0x0000ea6803098a70 0x166cea6803098a70 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0957f91 0x048638a0 0x0000ea6bf48638a0 0x166cea6bf48638a0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967e3d 0x04864070 0x0000ea6bf4864070 0x166cea6bf4864070 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4dd3 0x04865fb0 0x0000ea6bf4865fb0 0x166cea6bf4865fb0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0998027 0x04866780 0x0000ea6bf4866780 0x166cea6bf4866780 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7dd9 0x0486a600 0x0000ea6bf486a600 0x166cea6bf486a600 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d45 0x0486add0 0x0000ea6bf486add0 0x166cea6bf486add0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7e98 0x0486b5a0 0x0000ea6bf486b5a0 0x166cea6bf486b5a0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967d19 0x0486cd10 0x0000ea6bf486cd10 0x166cea6bf486cd10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e95 0x0486ec50 0x0000ea6bf486ec50 0x166cea6bf486ec50 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0997dba 0x0486f420 0x0000ea6bf486f420 0x166cea6bf486f420 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x8280cc32 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82816a15 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4c2e 0x04874240 0x0000ea6bf4874240 0x166cea6bf4874240 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7ea0 0x04874a10 0x0000ea6bf4874a10 0x166cea6bf4874a10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4b7e 0x048780c0 0x0000ea6bf48780c0 0x166cea6bf48780c0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0857dd8 0x0487a7d0 0x0000ea6bf487a7d0 0x166cea6bf487a7d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947ed6 0x0487a7d0 0x0000ea6bf487a7d0 0x166cea6bf487a7d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7f9f 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967f8e 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07d73 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977d81 0x0487b770 0x0000ea6bf487b770 0x166cea6bf487b770 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7d98 0x0487bf40 0x0000ea6bf487bf40 0x166cea6bf487bf40 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09d803b 0x0487c710 0x0000ea6bf487c710 0x166cea6bf487c710 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4ca6 0x0487d6b0 0x0000ea6bf487d6b0 0x166cea6bf487d6b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x8280cc32 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82816a15 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27dba 0x04880d60 0x0000ea6bf4880d60 0x166cea6bf4880d60 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4996 0x04881d00 0x0000ea6bf4881d00 0x166cea6bf4881d00 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0857f80 0x04883470 0x0000ea6bf4883470 0x166cea6bf4883470 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f50 0x04883c40 0x0000ea6bf4883c40 0x166cea6bf4883c40 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07e3b 0x04884410 0x0000ea6bf4884410 0x166cea6bf4884410 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977f31 0x048853b0 0x0000ea6bf48853b0 0x166cea6bf48853b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7db2 0x04885b80 0x0000ea6bf4885b80 0x166cea6bf4885b80 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4951 0x04886b20 0x0000ea6bf4886b20 0x166cea6bf4886b20 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27e54 0x0488a1d0 0x0000ea6bf488a1d0 0x166cea6bf488a1d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4bc5 0x0488b170 0x0000ea6bf488b170 0x166cea6bf488b170 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947d43 0x0488d0b0 0x0000ea6bf488d0b0 0x166cea6bf488d0b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0877f7a 0x0488e820 0x0000ea6bf488e820 0x166cea6bf488e820 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977e13 0x0488e820 0x0000ea6bf488e820 0x166cea6bf488e820 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4985 0x0488ff90 0x0000ea6bf488ff90 0x166cea6bf488ff90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08b7f0e 0x04893640 0x0000ea6bf4893640 0x166cea6bf4893640 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27dc7 0x04893640 0x0000ea6bf4893640 0x166cea6bf4893640 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4db1 0x04893e10 0x0000ea6bf4893e10 0x166cea6bf4893e10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4ea2 0x07d8b160 0x0000ea6d67d8b160 0x166cea6d67d8b160 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7eda 0x07d8b930 0x0000ea6d67d8b930 0x166cea6d67d8b930 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7c0e 0x07d8d0a0 0x0000ea6d67d8d0a0 0x166cea6d67d8d0a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4bdb 0x07d8efe0 0x0000ea6d67d8efe0 0x166cea6d67d8efe0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947de3 0x07d90750 0x0000ea6d67d90750 0x166cea6d67d90750 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0877fed 0x07d945d0 0x0000ea6d67d945d0 0x166cea6d67d945d0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4f9a 0x07d94da0 0x0000ea6d67d94da0 0x166cea6d67d94da0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07f5e 0x07d94da0 0x0000ea6d67d94da0 0x166cea6d67d94da0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7e2b 0x07d95570 0x0000ea6d67d95570 0x166cea6d67d95570 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4c6a 0x07d98c20 0x0000ea6d67d98c20 0x166cea6d67d98c20 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f27 0x07d99bc0 0x0000ea6d67d99bc0 0x166cea6d67d99bc0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7d09 0x07d9a390 0x0000ea6d67d9a390 0x166cea6d67d9a390 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937d1a 0x07d9bb00 0x0000ea6d67d9bb00 0x166cea6d67d9bb00 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x8280e345 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82816a16 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0878187 0x07d9e210 0x0000ea6d67d9e210 0x166cea6d67d9e210 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07ce8 0x07d9e210 0x0000ea6d67d9e210 0x166cea6d67d9e210 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c38a0 0x07d9e9e0 0x0000ea6d67d9e9e0 0x166cea6d67d9e9e0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4b25 0x07da2090 0x0000ea6d67da2090 0x166cea6d67da2090 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7c55 0x07da3030 0x0000ea6d67da3030 0x166cea6d67da3030 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937e6c 0x07da47a0 0x0000ea6d67da47a0 0x166cea6d67da47a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c36aa 0x07da7680 0x0000ea6d67da7680 0x166cea6d67da7680 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07c8e 0x07da7680 0x0000ea6d67da7680 0x166cea6d67da7680 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837f38 0x07da9d90 0x0000ea6d67da9d90 0x166cea6d67da9d90 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0917d84 0x07daad30 0x0000ea6d67daad30 0x166cea6d67daad30 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f39 0x07dab500 0x0000ea6d67dab500 0x166cea6d67dab500 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0977d15 0x07dabcd0 0x0000ea6d67dabcd0 0x166cea6d67dabcd0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d6c 0x07dac4a0 0x0000ea6d67dac4a0 0x166cea6d67dac4a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7f77 0x07dac4a0 0x0000ea6d67dac4a0 0x166cea6d67dac4a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x8280e346 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82816a16 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937f84 0x07dadc10 0x0000ea6d67dadc10 0x166cea6d67dadc10 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4bb4 0x07db0af0 0x0000ea6d67db0af0 0x166cea6d67db0af0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07cee 0x07db0af0 0x0000ea6d67db0af0 0x166cea6d67db0af0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e94 0x07db3200 0x0000ea6d67db3200 0x166cea6d67db3200 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d16 0x07db5910 0x0000ea6d67db5910 0x166cea6d67db5910 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937dd1 0x07db7080 0x0000ea6d67db7080 0x166cea6d67db7080 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7e42 0x07db87f0 0x0000ea6d67db87f0 0x166cea6d67db87f0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4ed0 0x07db9f60 0x0000ea6d67db9f60 0x166cea6d67db9f60 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07e15 0x07db9f60 0x0000ea6d67db9f60 0x166cea6d67db9f60 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0977f69 0x07dbaf00 0x0000ea6d67dbaf00 0x166cea6d67dbaf00 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e60 0x07dbc670 0x0000ea6d67dbc670 0x166cea6d67dbc670 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4bfb 0x07dbed80 0x0000ea6d67dbed80 0x166cea6d67dbed80 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837eb4 0x055a8c50 0x0000ea6d79a8c844 0x166cea6d755a8c50 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7dba 0x055a8c50 0x0000ea6d755a8c50 0x166cea6d755a8c50 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e6d 0x055a9420 0x0000ea6d755a9420 0x166cea6d755a9420 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc0907bef 0x055aa3c0 0x0000ea6d755aa3c0 0x166cea6d755aa3c0 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08b7ed5 0x055acad0 0x0000ea6d755acad0 0x166cea6d755acad0 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc0887eb6 0x055ae240 0x0000ea6d755ae240 0x166cea6d755ae240 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4da2 0x055aea10 0x0000ea6d755aea10 0x166cea6d755aea10 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7ff1 0x055b0180 0x0000ea6d755b0180 0x166cea6d755b0180 0x166cea6d79a8c844

 :
 :

MERGE Data Link (20503): bad timestamp  3 3 0xc0da81a0 0x0bb28852 0x0000ee992bb28852 0x166cee992bb28852 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f4811f 0x0bb29022 0x0000ee992bb29022 0x166cee992bb29022 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc805b 0x0bb297f2 0x0000ee992bb297f2 0x166cee992bb297f2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d38174 0x0bb2af62 0x0000ee992bb2af62 0x166cee992bb2af62 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c38233 0x0bb2ede2 0x0000ee992bb2ede2 0x166cee992bb2ede2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f68019 0x0bb2f5b2 0x0000ee992bb2f5b2 0x166cee992bb2f5b2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0x83807863 0x0bb30fca 0x0000ee992bb30fca 0x166cee992bb30fca 0x166cee992d6fc8b2
MERGE Data Link (20502): bad timestamp  2 3 0xc0877f3c 0x0e20b4a0 0x0000eeae2e20b4a0 0x166ceeae2e20b4a0 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7bc6 0x0e20b4a0 0x0000eeae2e20b4a0 0x166ceeae2e20b4a0 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e12 0x0e20bc70 0x0000eeae2e20bc70 0x166ceeae2e20bc70 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7df4 0x0e20bc70 0x0000eeae2e20bc70 0x166ceeae2e20bc70 0x166ceeae40e4c770

Earliest & latest timestamps from latest batch = Tuesday, March 16, 2021 7:48:18.767 PM - Tuesday, March 16, 2021 9:06:38.184 PM this evening
 
21.17 FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.5M data items/s
      TapeServer OK - 14Mb/s
Attachment 1: Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-16_Statistics_aidas-gsi.png
Screenshot_2021-03-16_Statistics_aidas-gsi.png
Attachment 3: 82.png
82.png
Attachment 4: Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-16_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-16_Statistics_aidas-gsi(1).png
Screenshot_2021-03-16_Statistics_aidas-gsi(1).png
Attachment 6: 83.png
83.png
  201   Mon Mar 15 11:47:51 2021 TDMonday 15 March
12.48 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

12.50 System wide checks

	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20     18      5      4      2      2      2      2      3      4     10   : 54576
aida02 :     26     18     16     10      6      4      2      4      2      3      6   : 37048
aida03 :     18     10      4      3      2      3      2      4      2      3      6   : 36152
aida04 :     13      9      4      2      2      2      2      4      1      4     15   : 73852
aida05 :     46     72     70     54     52     48     21     19      7      4      7   : 72216
aida06 :     32     10      4     10      0      2      2      4      3      2      5   : 31056
aida07 :     19      7      5      3      2      3      3      3      3      3      6   : 36916
aida08 :     20     14      2      3      1      3      1      4      3      3      6   : 36864
aida09 :      3      6      3      3      2      4      1      2      3      3      6   : 35916
aida10 :     23     11      3      3      1      3      3      2      3      3      6   : 36356
aida11 :      2      5      6      2      2      4      2      3      2      3      6   : 35664
aida12 :     25     10      5      2      2      2      2      3      3      3      6   : 36548


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 14Mb/s

12.55 Rate spectra - attachments 4 & 5

      Merger server error messages since most recent check

MERGE Data Link (20503): bad timestamp  3 3 0xc0d582ee 0x04f02352 0x000064aa97efd660 0x166c64aa94f02352 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e680cf 0x04f032f2 0x000064aa94f032f2 0x166c64aa94f032f2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c68215 0x04f03ac2 0x000064aa94f03ac2 0x166c64aa94f03ac2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd81f1 0x04f061d2 0x000064aa94f061d2 0x166c64aa94f061d2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0eb812a 0x04f07942 0x000064aa94f07942 0x166c64aa94f07942 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0f68041 0x04f07942 0x000064aa94f07942 0x166c64aa94f07942 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83806b78 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83811ea0 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c183d5 0x04f0aff2 0x000064aa94f0aff2 0x166c64aa94f0aff2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d58220 0x04f0b7c2 0x000064aa94f0b7c2 0x166c64aa94f0b7c2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e68097 0x04f0bf92 0x000064aa94f0bf92 0x166c64aa94f0bf92 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c682e7 0x04f0c762 0x000064aa94f0c762 0x166c64aa94f0c762 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cf7ff7 0x04f0cf32 0x000064aa94f0cf32 0x166c64aa94f0cf32 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd817e 0x04f0ee72 0x000064aa94f0ee72 0x166c64aa94f0ee72 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e180e5 0x04f105e2 0x000064aa94f105e2 0x166c64aa94f105e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d181fd 0x04f10db2 0x000064aa94f10db2 0x166c64aa94f10db2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d78072 0x04f11582 0x000064aa94f11582 0x166c64aa94f11582 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c17fff 0x04f14462 0x000064aa94f14462 0x166c64aa94f14462 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c68234 0x04f15bd2 0x000064aa94f15bd2 0x166c64aa94f15bd2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c780f9 0x04f17342 0x000064aa94f17342 0x166c64aa94f17342 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98277 0x04f17b12 0x000064aa94f17b12 0x166c64aa94f17b12 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d28174 0x04f17b12 0x000064aa94f17b12 0x166c64aa94f17b12 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d5812d 0x04f182e2 0x000064aa94f182e2 0x166c64aa94f182e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd8112 0x04f182e2 0x000064aa94f182e2 0x166c64aa94f182e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83806b78 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83811ea0 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d98224 0x04f18ab2 0x000064aa94f18ab2 0x166c64aa94f18ab2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e68096 0x04f19a52 0x000064aa94f19a52 0x166c64aa94f19a52 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c080e1 0x04f1a222 0x000064aa94f1a222 0x166c64aa94f1a222 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d18282 0x04f1a222 0x000064aa94f1a222 0x166c64aa94f1a222 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c38198 0x04f1a9f2 0x000064aa94f1a9f2 0x166c64aa94f1a9f2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cb8187 0x04f1b1c2 0x000064aa94f1b1c2 0x166c64aa94f1b1c2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc8178 0x04f1b992 0x000064aa94f1b992 0x166c64aa94f1b992 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c78063 0x04f1ffe2 0x000064aa94f1ffe2 0x166c64aa94f1ffe2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e980e1 0x04f1ffe2 0x000064aa94f1ffe2 0x166c64aa94f1ffe2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98129 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d28107 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0eb80e2 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660

18.47 FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 14Mb/s

      No new merger error messages since 12.55 today

          
Attachment 1: Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-15_Statistics_aidas-gsi.png
Screenshot_2021-03-15_Statistics_aidas-gsi.png
Attachment 3: 60.png
60.png
Attachment 4: Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 7: Screenshot_2021-03-15_Statistics_aidas-gsi(1).png
Screenshot_2021-03-15_Statistics_aidas-gsi(1).png
Attachment 8: 61.png
61.png
  200   Sun Mar 14 17:35:05 2021 TD, OHSunday 14 March - Pulser Walkthrough
18.01 Pulser walkthrough at end of run S452 file S452/R57_0 ... R57_28

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      BNC PB-5 settings changed by terminal command line (see attachments 1, 2 & 3)
      using the commands

      set rep rate 25
      set attenuation 5
      set amplitude 10
      set amplitude 9
      :
      :
      set amplitude 1
      set attenuation 1
      set rep rate 2

      Example n+n LEC spectrum - see attachment 4

18.27 R57 ends OK file R57_28
Attachment 1: 52.png
52.png
Attachment 2: 51.png
51.png
Attachment 3: 40.png
40.png
Attachment 4: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
  199   Sun Mar 14 08:16:00 2021 OH PJCSSunday 14th March 2021
> 08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151
> 
> Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
> It is also the least noisy FEE in the back DSSD
> 
> MERGE Data Link (20510): bad timestamp  10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
> MERGE Data Link (20510): bad timestamp  10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262
> 
> 09:01 System wide checks
> 
> 		 Base 		Current 	Difference
> aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
> White Rabbit error counter test result: Passed 11, Failed 1
> 
> Understand the status reports as follows:-
> Status bit 3 : White Rabbit decoder detected an error in the received data
> Status bit 2 : Firmware registered WR error, no reload of Timestamp
> Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
> 
> 			 Base 		Current 		Difference
> aida07 fault 	 0x2 : 	 0x5 : 	 3  
> FPGA Timestamp error counter test result: Passed 11, Failed 1
> If any of these counts are reported as in error
> The ASIC readout system has detected a timeslip.
> That is the timestamp read from the time FIFO is not younger than the last
> 
>      Statistics - attachment 1
>      Temperature - attachment 2
>      Bias and leakage current - attachment 3
It might be worth considering decreasing the time between the flushing of slower FEEs. Is the Merger waiting for the slower data and causing ‘deadtime’ in the faster ones. Perhaps consider this as a 
balancing across the system? 
  198   Sun Mar 14 07:04:30 2021 OHSunday 14th March 2021
08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151

Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
It is also the least noisy FEE in the back DSSD

MERGE Data Link (20510): bad timestamp  10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
MERGE Data Link (20510): bad timestamp  10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262

09:01 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Statistics - attachment 1
     Temperature - attachment 2
     Bias and leakage current - attachment 3

11:18 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c78 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics - attachment 4
     Temperatures - attachment 5
     Bias and leakage - attachment 6


13.14 DAQ continues OK - file R56_285

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

13.18 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c78 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      8      3      4      2      2      2      2      3      4     10   : 54492
aida02 :      0      2      3      0      2      3      2      3      3      3      6   : 36416
aida03 :     18      9      1      2      0      4      1      3      3      3      6   : 36320
aida04 :      2      3      3      3      1      4      3      2      2      4     15   : 74224
aida05 :     26     72     72     54     53     49     22     18      7      4      7   : 72104
aida06 :      7      5      5     10      0      2      2      4      3      2      5   : 30932
aida07 :      2      4      2      1      1      3      2      3      2      3      7   : 39464
aida08 :     25      9      4      2      1      2      1      4      3      3      6   : 36716
aida09 :      4     11      3      5      2      4      1      2      3      3      6   : 36024
aida10 :     18      5      2      1      0      5      2      2      3      3      6   : 36144
aida11 :     25     10      4      1      1      4      2      3      2      3      6   : 35668
aida12 :     26      7      1      1      1      3      2      3      3      3      6   : 36496

13.19 FEE64 temperatures OK - attachment 7
      Good events stats OK - attachment 8
      Bias & leakage currents OK - attachment 9
      Merger 4.5M items/s
      Tape Server 14Mb/s
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      All histograms zero'd


16.00 DAQ contiues R56_361

       System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c79 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      8      6      4      2      2      2      2      3      4     10   : 54524
aida02 :      1      3      4      2      1      2      1      4      3      3      6   : 36572
aida03 :      2      3      2      3      1      3      2      2      3      3      6   : 35936
aida04 :     19      7      2      3      1      4      3      3      1      4     15   : 73796
aida05 :     34     68     72     54     53     49     22     18      7      4      7   : 72104
aida06 :     11     10      4     10      0      2      2      4      3      2      5   : 30972
aida07 :      1      3      2      2      2      2      2      3      2      3      7   : 39420
aida08 :     24     10      5      2      1      3      1      4      3      3      6   : 36864
aida09 :     20      8      3      5      2      3      2      3      3      3      6   : 36704
aida10 :      2      3      2      3      1      3      3      2      3      3      6   : 36192
aida11 :     17     10      4      0      1      4      2      3      2      3      6   : 35604
aida12 :     20      6      3      2      1      3      2      2      2      4      6   : 37040


15.05 FEE64 temperatures OK - attachment 10
      Good events stats OK - attachment 11
      Bias & leakage currents OK - attachment 12
      Merger 4.6M items/s
      Tape Server 14Mb/s
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      Rate spectra - attachments 13 & 14
      p+n junction HEC spectra - attachment 15

17.33 end of S452 run file R56_397
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      BNC PB-5 config - attachment 16
      CAEN 8xxx NIM bin config - attachment 17
Attachment 1: 210314_0858_Stats.png
210314_0858_Stats.png
Attachment 2: 210314_0859_Temp.png
210314_0859_Temp.png
Attachment 3: 210314_0900_Bias.png
210314_0900_Bias.png
Attachment 4: 210314_1116_Stats.png
210314_1116_Stats.png
Attachment 5: 210314_1116_Temp.png
210314_1116_Temp.png
Attachment 6: 210314_1117_Bias.png
210314_1117_Bias.png
Attachment 7: Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi.png
Attachment 8: Screenshot_2021-03-14_Statistics_aidas-gsi.png
Screenshot_2021-03-14_Statistics_aidas-gsi.png
Attachment 9: 34.png
34.png
Attachment 10: Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 11: Screenshot_2021-03-14_Statistics_aidas-gsi(1).png
Screenshot_2021-03-14_Statistics_aidas-gsi(1).png
Attachment 12: 35.png
35.png
Attachment 13: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi.png
Attachment 14: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(1).png
Attachment 15: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 16: 40.png
40.png
Attachment 17: 41.png
41.png
Attachment 18: 51.png
51.png
Attachment 19: 52.png
52.png
Attachment 20: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
  197   Sat Mar 13 23:08:08 2021 CA14th March 00:00-08:00
00:08 DAQ continues ok - writing to file R46_221

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a4 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      6      2      5      1      3      2      2      3      4     10   : 54492
aida02 :     12      7      3      4      3      3      3      2      3      3      6   : 36440
aida03 :      4      4      2      4      2      3      2      2      3      3      6   : 36048
aida04 :     18      9      4      2      1      3      2      4      1      4     15   : 73936
aida05 :     31     11      8      7      6      4      2     18      7      4      7   : 55220
aida06 :     36      8     18      4      4      2      1      4      3      2      5   : 31088
aida07 :     26      7      4      3      1      2      1      3      3      3      6   : 36224
aida08 :     25     12      5      4      4      3      2      3      3      3      6   : 36884
aida09 :      2      9      4      2      2      2      2      3      3      3      6   : 36432
aida10 :     11      3      4      6      4      3      1      3      3      3      6   : 36548
aida11 :     21      8      5      0      1      3      3      3      2      3      6   : 35748
aida12 :     10      6      6      2      4      4      1      3      3      3      6   : 36600

00:14 FEE64 Temperatures ok - attachment 1
      Good event statistics ok - attachment 2
      Detector bias/leakage currents ok - attachment 3

00:17 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

01:30 beam off

01:31 beam back

02:19 system wide checks

      same as entry at 00:09, *except*

 		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a5 : 	 5  
White Rabbit error counter test result: Passed 11, Failed 1

02:22 FEE64 Temperatures ok - attachment 4
      Good event statistics ok - attachment 5
      Detector bias/leakage currents ok - attachment 6

02:30 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

      no further bad timestamp errors in NewMerger terminal

02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7

02:50 ... and sure enough lose connection to aida07 after checking stats

      DAQ stop, Merger/TapeServer closed

      telnet aida07 -> sent reboot command

      mounted ok, reset DAQ, TapeService and Merge

      restarted DATA forwarding to MBS

      all looks to be ok now -> writing to R56

03:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	        Base 	Current Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5      3      7      2      2      2      2      3      4     10   : 54508
aida02 :      2      7      3      2      2      3      1      4      3      3      6   : 36784
aida03 :      8      8      1      3      2      3      1      3      3      3      6   : 36304
aida04 :     20      9      2      3      0      3      2      4      1      4     15   : 73880
aida05 :     51     74     70     54     53     49     22     18      7      4      7   : 72188
aida06 :     15     15      3      9      1      2      2      4      3      2      5   : 31044
aida07 :     24      6      2      3      3      4      2      3      2      3      7   : 39888
aida08 :     19     15      1      2      1      3      1      4      3      3      6   : 36820
aida09 :     18      4      2      7      3      4      1      3      3      3      6   : 36648
aida10 :     20      7      1      1      1      3      3      2      3      3      6   : 36216
aida11 :     25      7      5      3      4      2      1      4      2      3      6   : 35916
aida12 :      3      3      2      2      3      2      3      2      3      3      6   : 36164

03:18 FEE64 Temperatures ok - attachment 8
      Good event statistics ok - attachment 9
      Detector bias/leakage currents ok - attachment 10

      Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s
      
03:29 more bad timestamp error messages - attachment 11

03:52 analyser output on R56_35 - attachment 12

      no timewarps

03:53 DAQ continues ok, writing to file R56_41

05:18 system wide checks:

      same as entry at 03:08, *but* with additional WR decoder status error

Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

05:19 FEE64 Temperatures ok - attachment 14
      Good event statistics ok - attachment 13
      Detector bias/leakage currents ok - attachment 15

      Merger ok - 4.4M items/s
      TapeServer ok - 15 MB/s

05:25 rate spectra - attachment 16

05:33 AIDA implant XY hit patterns - attachments 17/18/19

05:56 BuTIS interface control set to 0x3 at 04.52 UTC

07:06 system wide checks - same error messages as entry at 05:18

07:08 FEE64 Temperatures ok - attachment 20

      good event statistics ok - attachment 21

      detector bias/leakage currents ok - attachment 22

      Merger ok - 4.3M items/s
      TapeServer ok - 15 MB/s

07:22 no bad timestamp errors since 02:30 UTC
Attachment 1: Screenshot_from_2021-03-13_23-14-33.png
Screenshot_from_2021-03-13_23-14-33.png
Attachment 2: Screenshot_from_2021-03-13_23-14-57.png
Screenshot_from_2021-03-13_23-14-57.png
Attachment 3: Screenshot_from_2021-03-13_23-15-12.png
Screenshot_from_2021-03-13_23-15-12.png
Attachment 4: Screenshot_from_2021-03-14_01-22-56.png
Screenshot_from_2021-03-14_01-22-56.png
Attachment 5: Screenshot_from_2021-03-14_01-29-35.png
Screenshot_from_2021-03-14_01-29-35.png
Attachment 6: Screenshot_from_2021-03-14_01-29-52.png
Screenshot_from_2021-03-14_01-29-52.png
Attachment 7: Screenshot_from_2021-03-14_01-40-57.png
Screenshot_from_2021-03-14_01-40-57.png
Attachment 8: Screenshot_from_2021-03-14_02-16-28.png
Screenshot_from_2021-03-14_02-16-28.png
Attachment 9: Screenshot_from_2021-03-14_02-17-01.png
Screenshot_from_2021-03-14_02-17-01.png
Attachment 10: Screenshot_from_2021-03-14_02-17-32.png
Screenshot_from_2021-03-14_02-17-32.png
Attachment 11: Screenshot_from_2021-03-14_02-29-29.png
Screenshot_from_2021-03-14_02-29-29.png
Attachment 12: R56_35
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261409816 ( 1907248.6 Hz)
        Other data format:     510184 (    3722.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2488 (      18.2 Hz)
                               RESUME:       2487 (      18.1 Hz)
                              SYNC100:        511 (       3.7 Hz)
                              WR48-63:        511 (       3.7 Hz)
                           FEE64 disc:     503410 (    3672.9 Hz)
                             MBS info:        777 (       5.7 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      60911 (     444.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      137.061 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.725              132.339
  2                3.132              135.291
  3                6.180              132.876
  4               12.844              128.581
  5                8.386              132.070
  6                1.216              132.607
  7                1.093              115.159
  8               50.447              129.654
  9               19.940              134.755
 10                0.568              132.339
 11               51.844              133.681
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22917695      26526          0          0        878        878         65         65      23863        777          0       8590
  1   26593258      35084          0          0         13         13         51         51      34956          0          0      10875
  2   25634005     337148          0          0         16         16         76         76     336964          0          0       6926
  3   28548879       3343          0          0         34         34         27         27       3221          0          0       1214
  4    7607641      18967          0          0        663        663         14         14      17613          0          0       6303
  5   19366719      25239          0          0        305        304         24         24      24582          0          0       8037
  6   18601227      17453          0          0          3          3         18         18      17411          0          0       6563
  7   10076045       2592          0          0          4          4         24         24       2536          0          0        908
  8   30026924       8042          0          0        217        216         34         34       7541          0          0       2274
  9   30858440      18084          0          0        113        113         88         88      17682          0          0       4590
 10    9946552      15820          0          0          3          3         46         46      15722          0          0       4264
 11   31232431       1886          0          0        239        240         44         44       1319          0          0        367
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:13611.407s (    2.351 blocks/s,   0.147 Mb/s)
Attachment 13: Screenshot_from_2021-03-14_04-20-07.png
Screenshot_from_2021-03-14_04-20-07.png
Attachment 14: Screenshot_from_2021-03-14_04-20-34.png
Screenshot_from_2021-03-14_04-20-34.png
Attachment 15: Screenshot_from_2021-03-14_04-20-50.png
Screenshot_from_2021-03-14_04-20-50.png
Attachment 16: Screenshot_from_2021-03-14_04-25-41.png
Screenshot_from_2021-03-14_04-25-41.png
Attachment 17: Screenshot_from_2021-03-14_04-32-03.png
Screenshot_from_2021-03-14_04-32-03.png
Attachment 18: Screenshot_from_2021-03-14_04-32-32.png
Screenshot_from_2021-03-14_04-32-32.png
Attachment 19: Screenshot_from_2021-03-14_04-33-14.png
Screenshot_from_2021-03-14_04-33-14.png
Attachment 20: Screenshot_from_2021-03-14_06-10-39.png
Screenshot_from_2021-03-14_06-10-39.png
Attachment 21: Screenshot_from_2021-03-14_06-11-19.png
Screenshot_from_2021-03-14_06-11-19.png
Attachment 22: Screenshot_from_2021-03-14_06-11-38_-_1.png
Screenshot_from_2021-03-14_06-11-38_-_1.png
  196   Sat Mar 13 17:00:49 2021 TDSaturday 13 March 18.00-00.00 UTC+1
18.00 DAQ continues OK - file R46_64

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

18.02 System wide checks OK *except*

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 16Mb/s

      All histograms zero'd

17.15 Most recent messages in merger server terminal session

MERGE Data Link (2322): bad timestamp  6 3 0xc1b57e35 0x0ba1685c 0x0000ef3fbba1685c 0x166bef3fbba1685c 0x166bef3fbba1879c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a77dc2 0x0ba1702c 0x0000ef3fbba1702c 0x166bef3fbba1702c 0x166bef3fbba1879c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bf7f66 0x0ba1702c 0x0000ef3fbba1702c 0x166bef3fbba1702c 0x166bef3fbba1879c
Transfer Error - : Broken pipe
Error 32
1: send() failed:
TCP transfer library version 4.0
1: TCP socket send buffer was 16384 - now 249856
1: TCP socket receive buffer was 87380 - now 249856
1: TCP socket created OK - now connecting to localhost port 10305
1: Connected to localhost port 10305


22.05 DAQ continues OK - file R46_168

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a2 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     18      6      7      5      1      3      1      3      3      4     10   : 54856
aida02 :     19      6      4      3      4      4      2      3      3      3      6   : 36892
aida03 :     22      8      1      2      1      2      1      3      3      3      6   : 36136
aida04 :     24      8      4      2      1      3      2      4      1      4     15   : 73952
aida05 :     37     14      9      8      7      3      2     18      7      4      7   : 55252
aida06 :     28     12     16      4      4      2      1      4      3      2      5   : 31056
aida07 :     28      9      2      2      2      2      1      3      3      3      6   : 36248
aida08 :     21     12      6      4      3      4      2      3      3      3      6   : 36948
aida09 :     18      5      2      1      1      3      2      2      3      3      6   : 35952
aida10 :     17      8      4      5      3      3      1      3      3      3      6   : 36516
aida11 :     21      8      5      0      0      4      3      3      2      3      6   : 35812
aida12 :     13     11      3      3    


      FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.4M data items/s
      TapeServer OK - 15Mb/s

22.17 Rate spectra - attachments 7 & 8
      p+n junction HEC spectra - attachment 9

      Merger server error messages since 17.15

MERGE Data Link (2316): bad timestamp  0 3 0x80300000 0x0164e7f8 0x000000000164e7f8 0x166200000164e7f8 0x166bffffffa7654e
MERGE Data Link (2324): bad timestamp  8 3 0x88300000 0x0534e25c 0x000000000534e25c 0x166200000534e25c 0x166bffffed8383fc
MERGE Data Link (2322): bad timestamp  6 3 0xc1a07f16 0x0070e9cc 0x000001e74070e9cc 0x166c01e74070e9cc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b37895 0x0070e9cc 0x000001e74070e9cc 0x166c01e74070e9cc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc19f7e41 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a27e22 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b47399 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ab7dad 0x0070f96c 0x000001e74070f96c 0x166c01e74070f96c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b564f9 0x0070f96c 0x000001e74070f96c 0x166c01e74070f96c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b60177 0x0071013c 0x000001e74071013c 0x166c01e74071013c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a57e95 0x00e2208c 0x000001e700e2208c 0x166c01e700e2208c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b81491 0x00e2208c 0x000001e700e2208c 0x166c01e700e2208c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ac7e4f 0x00e2285c 0x000001e700e2285c 0x166c01e700e2285c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b90000 0x00e2285c 0x000001e700e2285c 0x166c01e700e2285c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ba0c64 0x00e2302c 0x000001e700e2302c 0x166c01e700e2302c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bb6ea8 0x00e237fc 0x000001e700e237fc 0x166c01e700e237fc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a87dc6 0x00e23fcc 0x000001e700e23fcc 0x166c01e700e23fcc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bc0641 0x00e23fcc 0x000001e700e23fcc 0x166c01e700e23fcc 0x166c01e74071090c
Attachment 1: Screenshot_2021-03-13_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-13_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-13_Statistics_aidas-gsi.png
Screenshot_2021-03-13_Statistics_aidas-gsi.png
Attachment 3: 32.png
32.png
Attachment 4: Screenshot_2021-03-13_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-13_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-13_Statistics_aidas-gsi(1).png
Screenshot_2021-03-13_Statistics_aidas-gsi(1).png
Attachment 6: 33.png
33.png
Attachment 7: Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-13_Spectrum_Browser_aidas-gsi(2).png
  195   Sat Mar 13 07:08:52 2021 CA, TD, LSMarch 13th 08:00 - 17:00
ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


08:09 System wide checks ok *except*

      aida09 fails clock check

      aida09 calibration failed

08:14 FEE64 temperatures ok - attachment 1

      good event statistics ok - attachment 2

      detector bias/leakage current ok - attachment 3

08:22 Merger 4.3M items/s

      Tapeserver 14 MB/s

08:24 DAQ ok, writing to file R43_45

08:39 rate spectra - attachment 4

      aida09 spectrum not showing, however continues to collect statistics ok

10:03 System wide checks ok *except*

      aida09 fails clock check

      aida09 calibration failed

      FEE64 temperatures ok - attachment 5

      good event statistics ok - attachment 6

      detector bias/leakage current ok - attachment 7

10:10 Merger 4.5M items/s

      Tapeserver 14 MB/s

      data forwarding to MBS ok

10:11 writing to file R43_91

12.00 (LS)
      System wide checks:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      		         Base 		Current 	Difference
      aida07 fault 	 0x829e : 	 0x829f : 	 1  
      White Rabbit error counter test result: Passed 11, Failed 1
      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      no FGPA timestamp errors all passed

      Statistics (attachment8)
      Spectra rate (attachment9), aida09 still does not show histogram, but on run control are enabled for all fees. still 
      showing good events and other information (temperatures etc)
      FEE temps (attachment10)
      Leakage currents written to sheets (attachment11)
      Merger~4.6M items/s
      Tapeserver~14MB/s

12.36 bunch of bad timestamp errors in the new merger terminal (attachment 12) around file R43_152
      this error is still occurring at 12.45 will continue to monitor, all other system checks are reporting as normal


13.50 beam down file R43_183
      beam back a couple of minutes later

14.00 System wide checks:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      no white rabbit or FPGA errors all passed

      Statistics (attachment13)
      Spectra rate (attachment14)
      FEE temps (attachment15)
      Leakage currents written to sheets (attachment16)
      Merger~4.6M items/s
      Tapeserver~14MB/s  
      Writing to MBS okay, still seeing bad timestamp errors in the new merger terminal continuing to monitor and update 
      statistics tab every 20 to 30 minutes 

14.12 Analysed files R43_185,186,187 from timestamp error in ucesb, but see no timewarps  

14.30 have not seen any bad timestamp errors in new merger terminal for a while, unsure if something has been changed which 
      has stopped them

15.25 beam down
15.53 beam back

16.00 moved back to writing to /media/SecondDrive which should last until the end of the experiment
      now writing R46

16.10 System wide checks okay except:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment17)
      Spectra rate (attachment18)
      FEE temps (attachment19)
      Leakage currents written to sheets (attachment20)
      Merger~4.7M items/s
      Tapeserver~14MB/s  
      Writing to MBS okay
Attachment 1: Screenshot_from_2021-03-13_07-12-39.png
Screenshot_from_2021-03-13_07-12-39.png
Attachment 2: Screenshot_from_2021-03-13_07-13-15.png
Screenshot_from_2021-03-13_07-13-15.png
Attachment 3: Screenshot_from_2021-03-13_07-13-46.png
Screenshot_from_2021-03-13_07-13-46.png
Attachment 4: Screenshot_from_2021-03-13_07-38-52.png
Screenshot_from_2021-03-13_07-38-52.png
Attachment 5: Screenshot_from_2021-03-13_09-04-04.png
Screenshot_from_2021-03-13_09-04-04.png
Attachment 6: Screenshot_from_2021-03-13_09-04-28.png
Screenshot_from_2021-03-13_09-04-28.png
Attachment 7: Screenshot_from_2021-03-13_09-04-50.png
Screenshot_from_2021-03-13_09-04-50.png
Attachment 8: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 11: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 12: 1200_NewMerger.PNG
1200_NewMerger.PNG
Attachment 13: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 14: 1400_SpectraRate.PNG
1400_SpectraRate.PNG
Attachment 15: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 16: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 17: 1600_Statistics.PNG
1600_Statistics.PNG
Attachment 18: 1600_SpectraRate.PNG
1600_SpectraRate.PNG
Attachment 19: 1600_FEETemps.PNG
1600_FEETemps.PNG
Attachment 20: 1600_LeakageCurrents.PNG
1600_LeakageCurrents.PNG
  194   Fri Mar 12 23:56:00 2021 OHSaturday March 13th 00:00-08:00
24:00 Joined in with TD to debug the dropout of AIDA05 - See https://elog.ph.ed.ac.uk/DESPEC/193
      After both powercycles and telnet reboots we were still seeing 0 in the statistics but did note that the counter was > 0
      The last check we did was a restart of the merger server. This solved the issue.
      It is likely that the link to the FEE was dropped but not re-established upon the resets.
      There was no error message that this was seen. From now on I recommend refreshing the statistics ever 30 minutes.

01:29 System wide check
N.B. May not have reset baselines after reset
WR fault
		 Base 		Current 	Difference
aida01 fault 	 0x1ad : 	 0x1af : 	 2  
aida02 fault 	 0x4434 : 	 0x4436 : 	 2  
aida03 fault 	 0xcf95 : 	 0xcf99 : 	 4  
aida04 fault 	 0x25aa : 	 0x25ae : 	 4  
aida05 fault 	 0x4b4 : 	 0x4b7 : 	 3  
aida06 fault 	 0x834 : 	 0x837 : 	 3  
aida07 fault 	 0x146d : 	 0x1472 : 	 5  
aida08 fault 	 0xf4a8 : 	 0xf4ab : 	 3  
aida09 fault 	 0x4f6f : 	 0x4f73 : 	 4  
aida10 fault 	 0x7504 : 	 0x7506 : 	 2  
aida11 fault 	 0x26fa : 	 0x26fc : 	 2  
aida12 fault 	 0x2858 : 	 0x285b : 	 3  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Check
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x4 : 	 4  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

01:42 Statistics - attachment 1
      Temperature - attachment 2
      Bias and leakage current - attachment 3

03:57 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x1472 : 	 0x1473 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x4 : 	 0xc : 	 8  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Statistics - attachment 4
      Temp - attachment 5
      Bias - attachment 6

05:27 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x1472 : 	 0x1474 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x4 : 	 0x16 : 	 18  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


     Statistics - attachment 7
     Temp - attachment 8
     Bias - attachment 9

05:51 Looking into the bad timestamp messages in the merger e.g.
MERGE Data Link (28348): bad timestamp  5 3 0xc14f8415 0x00eea280 0x0000cd2e70eea280 0x166bcd2e70eea280 0x166bcd2e716c1f80

Looking in the merger source for the bad timestamp message:

            if (op->Time < LastTimeStamp) {

//     invalid time stamp 

               (*StatsMem[TSSEQERR])++;

               (*StatsMem[TSSEQERR+((LinkNum+1)*MAXCOUNTERS)])++;

               sprintf(message_buffer, "bad timestamp  %d %d 0x%08lx 0x%08lx 0x%016llx 0x%016llx 0x%016llx",LinkNum, link_table[LinkNum]->link_state, op->Data, op->Timestamp, INFO4, op->Time, LastTimeStamp);
      
                report_message(MSG_WARNING);   /***************************/

//              LastTimeStamp = op->Time;


So the message is generated when the merger detects a timewarp.

Took the first warning a data block of errors (The first instance of that particular `LastTimeStamp` and c alculated the time difference between the new timestamp and the last timestamp
The time and LastTimestamp were 0x166bccef3f424a50 0x166bccef4f4232e0 respectively
The time difference between them is 268429456 which is 268ms which seems quite a large difference


06:39 AIDA Crashed
      This time all FEEs were responsive but not showing any stats
      When stopping the DAQ all FEEs except aida06 stopped - attachment 10
      Did a reset of the DAQ and all recovered but no stats on aida06 - attachment 11
      Regained DAQ with a powercycle and a complete restart of the AIDA:8115 Merger and TaperServer

      It is worth noting that aida06 is connected to link 5 the data link which had been producing the bad merge messages overnight.
      We have now had it in aida05, aida6 and aida07. Could it be to do with the correlation scaler rate going into these FEES?

      Going through the var/log/messages on aida-3 aida06 rebooted itself at 06:37
      Mar 13 06:37:16 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Mar 13 06:37:18 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Mar 13 06:37:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)

Looking in /var/log/messages on aida06 no evidence of a reason why:
Mar 12 23:30:56 aida06 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: driver assigned major number 253
Mar 12 23:31:13 aida06 kernel: xaida: open:
Mar 12 23:31:14 aida06 kernel: AIDAMEM: aidamem_open:
Mar 12 23:33:06 aida06 kernel: xaida: open:
Mar 13 05:37:30 aida06 syslogd 1.4.2: restart.
Mar 13 05:37:30 aida06 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Mar 13 05:37:30 aida06 kernel: Using Xilinx Virtex440 machine description
Mar 13 05:37:30 aida06 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
Mar 13 05:37:30 aida06 kernel: Zone PFN ranges:
Attachment 1: 210313_0138_Stats.png
210313_0138_Stats.png
Attachment 2: 210313_0141_Temp.png
210313_0141_Temp.png
Attachment 3: 210313_0142_Bias.png
210313_0142_Bias.png
Attachment 4: 210313_0355_Stats.png
210313_0355_Stats.png
Attachment 5: 210313_0356_Bias.png
210313_0356_Bias.png
Attachment 6: 210313_0356_Temp.png
210313_0356_Temp.png
Attachment 7: 210313_0528_Stats.png
210313_0528_Stats.png
Attachment 8: 210313_0529_Temp.png
210313_0529_Temp.png
Attachment 9: 210313_0530_Bias.png
210313_0530_Bias.png
Attachment 10: 2210313_0640_AIDAissue.png
2210313_0640_AIDAissue.png
Attachment 11: 210313_0647_aida6MMissing.png
210313_0647_aida6MMissing.png
  193   Fri Mar 12 20:12:10 2021 TDFriday 12 March
21.12 DAQ continues OK - file R33_617

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

21.15 System wide checks


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x7a58 : 	 0x7a5a : 	 2  
aida06 fault 	 0x65bd : 	 0x65bf : 	 2  
aida07 fault 	 0xcdd6 : 	 0xcdda : 	 4  
aida08 fault 	 0x2ab5 : 	 0x2ab7 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      4      2      0      1      2      3      3      2      3     11   : 55956
aida02 :     21      4      1      2      2      4      2      2      1      4      7   : 40260
aida03 :     10      4      4      1      6      4      1      3      3      3      6   : 36648
aida04 :      7      2      4      0      2      3      2      4      1      4     15   : 73836
aida05 :      1      3      1      5      4      3      1      2      3      4      6   : 37964
aida06 :     12      3      5      2      1      3      3      3      2      4      7   : 41880
aida07 :      2      4      3      3      4      2      2      4      3      3      6   : 37048
aida08 :     17     13      2      3      2      3      2      3      2      3      7   : 39724
aida09 :      3      7      2      0      1      2      2      3      2      2      7   : 37284
aida10 :      2      5      3      2      2      3      1      3      2      3      7   : 39328
aida11 :     23     18      2      3      2      3      3      2      3      3      6   : 36460
aida12 :     18      7      3      1      1      4      2      2      2      3      7   : 39184

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.9M data items/s
      TapeServer OK - 16Mb/s

      All histograms zero'd

21.50 At some point between R33_610 (20.59) and R33_625 (21.30) aida05 stopped producing data (zero good events - see attachment 2)

      able to telnet to aida05 - no warnings/error messages in /var/log/messages

      DAQ STOP (all except aida05 stopped OK, aida05 remained GOING)

      issued aida05 reboot command via telnet command line

      DAQ RESET/SETUP/GO (all FEE64s GOING OK except aida05 - zero good events)

21.15 All system wide checks OK *except*

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


21.50 aida04 & aida05 statistics for comparison - see attachments 4 & 5
Attachment 1: Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 2: Screenshot_2021-03-12_Statistics_aidas-gsi(1).png
Screenshot_2021-03-12_Statistics_aidas-gsi(1).png
Attachment 3: 31.png
31.png
Attachment 4: Screenshot_2021-03-12_Statistics_aidas-gsi(2).png
Screenshot_2021-03-12_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-03-12_Statistics_aidas-gsi(3).png
Screenshot_2021-03-12_Statistics_aidas-gsi(3).png
  192   Fri Mar 12 10:09:19 2021 LS, CAFriday 12th March 11.00-
11.00(Germany) System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment1)
      Spectra rates (attachment2)
      FEE temps (attachment3)
      Leakage currents, written to google sheet (attachment4)  
      Merger~ 4.9M items/s
      Tapeserver ~17MB/s 

      In MBS control terminal, connection has been closed intentionally since this morning (file S452f160),
      AIDA has been taken out of the timesorter due to the high data rate, buffers were full
      AIDA cannot be seen in ucesb or Go4.

13.00 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment5)
      Spectra rates (attachment6)
      FEE temps (attachment7)
      Leakage currents, written to google sheet (attachment8)  
      Merger~ 5.1M items/s
      Tapeserver ~18MB/s 

      No timestamp related errors this shift

13.37 AIDA MBS control restarted R33_388
      System wide checks same as previous time

13.54 beam stopped for access,  file R33_396

13.57 seen recent batch of bad timestamp errors in new merger terminal (attachment9) should be around R33_397
      Analysed R33_395, 396, 397, 398 no timewarps
      
14.29 beam back R33_415

14.54 see more bad timestamps in new merger terminal (attachment10)

14.55 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment11)
      Spectra rates (attachment12)
      FEE temps (attachment13)
      Leakage currents, written to google sheet (attachment14)  
      Merger~ 5.0M items/s
      Tapeserver ~17MB/s 

16.05 AIDA included back into timesorter, AIDA scalers now seen in ucesb, file R33_463

16.15 CA takes over until 18:00

17:11 System wide checks:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0xcdd6 : 	 0xcdd7 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

17:13 FEE64 temps ok - attachment 15
      Statistics ok - attachment 16
      bias and leakage currents ok - attachment 17
Attachment 1: 1100_Statistics.PNG
1100_Statistics.PNG
Attachment 2: 1100_SpectraRate.PNG
1100_SpectraRate.PNG
Attachment 3: 1100_FEETemps.PNG
1100_FEETemps.PNG
Attachment 4: 1100_LeakageCurrents.PNG
1100_LeakageCurrents.PNG
Attachment 5: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 6: 1300_SpectraRate.PNG
1300_SpectraRate.PNG
Attachment 7: 1300_FEETemps.PNG
1300_FEETemps.PNG
Attachment 8: 1300_LeakageCurrents.PNG
1300_LeakageCurrents.PNG
Attachment 9: 1358_NewMerger.PNG
1358_NewMerger.PNG
Attachment 10: 1454_NewMerger.PNG
1454_NewMerger.PNG
Attachment 11: 1500_Statistics.PNG
1500_Statistics.PNG
Attachment 12: 1500_SpectraRate.PNG
1500_SpectraRate.PNG
Attachment 13: 1500_FEETemps.PNG
1500_FEETemps.PNG
Attachment 14: 1500_LeakageCurrents.PNG
1500_LeakageCurrents.PNG
Attachment 15: Screenshot_from_2021-03-12_16-18-13.png
Screenshot_from_2021-03-12_16-18-13.png
Attachment 16: Screenshot_from_2021-03-12_16-18-51.png
Screenshot_from_2021-03-12_16-18-51.png
Attachment 17: Screenshot_from_2021-03-12_16-21-21.png
Screenshot_from_2021-03-12_16-21-21.png
  191   Fri Mar 12 08:32:46 2021 TDFriday 12 March 08.00-
08.31 DAQ continues OK - file R33_263

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

08.32 No merger server error/warning messages since most recent restart c. 00.00 today


08.44 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x7a56 : 	 0x7a58 : 	 2  
aida06 fault 	 0x65bb : 	 0x65bd : 	 2  
aida07 fault 	 0xcdd1 : 	 0xcdd4 : 	 3  
aida08 fault 	 0x2ab3 : 	 0x2ab5 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     14      6      3      2      1      2      2      3      3      4     11   : 58904
aida02 :      6      4      1      1      1      2      2      4      2      3      7   : 39848
aida03 :      5      6      4      2      1      3      1      3      2      3      7   : 39300
aida04 :      4      1      2      2      3      3      2      4      1      4     15   : 73912
aida05 :      1      5      2      2      2      4      2      3      2      3      7   : 39692
aida06 :      7      2      2      3      1      3      3      3      2      4      7   : 41836
aida07 :      6      3      1      2      3      2      2      3      3      3      7   : 40512
aida08 :      8      5      2      2      1      3      1      3      2      3      7   : 39272
aida09 :      1      2      3      1      1      3      2      3      1      4      7   : 40484
aida10 :      2      6      3      2      1      2      2      2      3      3      7   : 39912
aida11 :      2      3      1      0      2      4      1      3      2      3      7   : 39344
aida12 :      2      3      4      2      0      3      3      3      2      3      7   : 39712


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 5.1M data items/s
      TapeServer OK - 16Mb/s

08.54 p+n junction strip HEC spectra - attachment 1
      Rate spectra - attachment 2

09.15 Burst of Merger server warning messages of type

MERGE Data Link (3547): bad timestamp  1 3 0x8128d6c8 0x05056328 0x00008d6c85056328 0x166b8d6c85056328 0x166b8d6d86da01ee

      also multiple

Warning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one 
MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in 
relayWarning: At least one MIDAS block missed in relayWarning: A

      The latter may reflect downstream problems writing MBS data
Attachment 1: Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-12_Statistics_aidas-gsi.png
Screenshot_2021-03-12_Statistics_aidas-gsi.png
Attachment 3: 30.png
30.png
Attachment 4: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi.png
  190   Thu Mar 11 23:07:23 2021 CAMarch 12th 00:00 - 08:00 shift
00:10 DAQ continues OK - file R30_372

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:17 System wide checks all OK

      FEE64 Temperatures ok - attachment 1

      Good event statistics ok - attachment 2

      detector bias and leakage currents ok - attachment 3

00:20 DESPEC on run 145

00:24 Merger ok ~45M items/s
      Tapeserver ok ~15MB/s

01:00 aida07 crashed

      recovered, but all FEE64 crash shortly after

      called OH - stop DAQ, TapeService, Merger

01:20 all FEE64 powercycled, AIDA restart

01:30 AIDA recovered, DAQ now running - writing to R33

      system wide checks ok *except*

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

01:52 FEE64 Temperatures ok - attachment 4

      -had to reload a few times to work, but otherwise ok

      Good event statistics - attachment 5

      -aida05 and aida08 running faster than before

      detector bias and leakage currents ok - attachment 6

02:04 writing to file R33_34

      Data forwarding to MBS ok

      AIDA ASIC settings ok

02:11 beam off

02:13 attempted to recalibrate aida07 and aida09 in FADC Align and Control - calibration still fails 

02:32 beam back - writing to file R33_48

04:30 DESPEC having issues with Go4 crashing

      ucesb reports AIDA/FATIMA/bplast timewarp events

      System wide checks:

      Clock error:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Calibration:
      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      White Rabbit:
       Base 		Current 	Difference
       aida07 fault 	 0xcdd1 : 	 0xcdd2 : 	 1  
      White Rabbit error counter test result: Passed 11, Failed 1

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      FEE64 Temperatures ok - attachment 7

      Good event statistics ok - attachment 8

      detector bias and leakage currents ok - attachment 9

04:52 analyzer output for R33_113 - attachment 10

      no timewarps

      aida06 dead time? (ignore idle time and rates)

05:02 Merger 5M data items/s

      TapeServer 17 MB/s

05:15 DESPEC believe issue with bplast TAMEX causing ucesb issues/Go4 crash

      They disable bplast and FATIMA TAMEX histograms in their online analysis - ucesb/go4 much more stable now

05:25 rates spectra - attachment 11

05:29 error message in MBS relay terminal - otherwise data forwarding to MBS ok - attachment 12

06:29 system wide checks:

      Clock error:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1

      Calibration:
      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2

      White Rabbit:
                         Base 		Current 	Difference
      aida05 fault 	 0x7a56 : 	 0x7a58 : 	 2  
      aida06 fault 	 0x65bb : 	 0x65bd : 	 2  
      aida07 fault 	 0xcdd1 : 	 0xcdd4 : 	 3  
      aida08 fault 	 0x2ab3 : 	 0x2ab5 : 	 2  
      White Rabbit error counter test result: Passed 8, Failed 4

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      rest ok

06:32 FEE64 Temperatures ok - attachment 13

      Good event statistics ok - attachment 14

      detector bias and leakage currents ok - attachment 15

06:41 Merger 5.1M data items/s

      TapeServer 17 MB/s

      Data forwarding to MBS ok

08:54 restart MBS relay, requested by NH

      













      

      
Attachment 1: Screenshot_from_2021-03-11_23-14-57_-_1.png
Screenshot_from_2021-03-11_23-14-57_-_1.png
Attachment 2: Screenshot_from_2021-03-11_23-15-31.png
Screenshot_from_2021-03-11_23-15-31.png
Attachment 3: Screenshot_from_2021-03-11_23-16-08.png
Screenshot_from_2021-03-11_23-16-08.png
Attachment 4: Screenshot_from_2021-03-12_00-55-09.png
Screenshot_from_2021-03-12_00-55-09.png
Attachment 5: Screenshot_from_2021-03-12_00-55-35.png
Screenshot_from_2021-03-12_00-55-35.png
Attachment 6: Screenshot_from_2021-03-12_00-58-59.png
Screenshot_from_2021-03-12_00-58-59.png
Attachment 7: Screenshot_from_2021-03-12_03-35-47.png
Screenshot_from_2021-03-12_03-35-47.png
Attachment 8: Screenshot_from_2021-03-12_03-36-35.png
Screenshot_from_2021-03-12_03-36-35.png
Attachment 9: Screenshot_from_2021-03-12_03-37-20.png
Screenshot_from_2021-03-12_03-37-20.png
Attachment 10: R33_113
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  260049813 ( 2219319.2 Hz)
        Other data format:    1870189 (   15960.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:      20863 (     178.0 Hz)
                               RESUME:      20864 (     178.1 Hz)
                              SYNC100:        437 (       3.7 Hz)
                              WR48-63:        437 (       3.7 Hz)
                           FEE64 disc:     442392 (    3775.5 Hz)
                             MBS info:    1385196 (   11821.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      32687 (     279.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      117.175 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1               49.525              113.280
  2               56.916              110.595
  3               47.492              115.964
  4               57.911              115.427
  5                7.968              116.501
  6       1615384952.728              108.448
  7               42.814              114.622
  8               80.186              113.011
  9               88.924               96.905
 10               21.378              111.132
 11               73.040              114.354
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   22118772      49791          0          0      18934      18933         32         32      11506        354          0       4012
  1   22984048      20152          0          0        166        167         43         43      19337        396          0       5880
  2   20394651     783427          0          0        163        164         37         37     255035     527991          0       3363
  3   22242991     614086          0          0        167        167         40         40       2080     611592          0        620
  4   21439332      55343          0          0        169        170         49         49       8463      46443          0       2902
  5   16079204     115814          0          0        279        279         29         29      22786      92412          0       7566
  6   22040349     143536          0          0        183        183         31         31     102212      40896          0       2700
  7   22240015      67179          0          0        150        150         58         58       1651      65112          0        545
  8   21453279       3177          0          0        179        179         23         23       2773          0          0        850
  9   22650021       4350          0          0        185        184         20         20       3941          0          0       1026
 10   24195088      12236          0          0        107        107         38         38      11946          0          0       3047
 11   22212063       1098          0          0        181        181         37         37        662          0          0        176
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:17230.426s (    1.857 blocks/s,   0.116 Mb/s)
Attachment 11: Screenshot_from_2021-03-12_04-23-51.png
Screenshot_from_2021-03-12_04-23-51.png
Attachment 12: Screenshot_from_2021-03-12_04-29-06.png
Screenshot_from_2021-03-12_04-29-06.png
Attachment 13: Screenshot_from_2021-03-12_05-34-02.png
Screenshot_from_2021-03-12_05-34-02.png
Attachment 14: Screenshot_from_2021-03-12_05-34-28_-_1.png
Screenshot_from_2021-03-12_05-34-28_-_1.png
Attachment 15: Screenshot_from_2021-03-12_05-34-51.png
Screenshot_from_2021-03-12_05-34-51.png
  189   Thu Mar 11 07:09:25 2021 OH, LSThursday 11th of March
08:00 System wide checks all ok *exctept aida07 and 10 on ADC calibration
      N.B. wrong date given to the screenshots by accident (It's early)
      Statistics - attachment 1
      FEE temperatures (AIDA10 unavailable) - attachment 2
      FEE temperature - attachment 3
      Bias and leakage currents ok - attachment 4

08:34 The FRS stopped receiving ions. AIDA crashed simultaneously with it.
      Have now recovered AIDA daq - Looking at the messages log AIDA04 restarted itself.
      Restarted but no stats in aida04
      It did not recover from a reset
      Telnet in and did a reboot command

09:10 Finally recovered issue was it was not linking to merger
      There is an options issue with aida12. The ASIC settings had become undefined and there are no histograms
      Manually set the ASIC settings for aida12. Rates look as expected after
      Will try to reset once beam is back
      

10:12 Problem with the DAQ so stopping AIDA while they work will switch MBS relay out to new version
      Analysis of R30_21 looks fine
      
10:38 System wide checks ok
      Statistics - attachment 5
      Temperature - attachment 6
      Bias and leakage - attachment 7

12.10 (LS)
      System wide checks all okay, no fails
      Statistics (attachment8)
      Rate spectra (attachment9)
      FEE temps (attament10)
      Leakage currents written to sheet (attachment11)
      Merger ~45M items/s
      Tapeserver ~14MB/s

13.45 no beam (around file R30_98)

14.00 System wide checks all okay, no fails
      Statistics (attachment12)
      FEE temps (attament13)
      Leakage currents written to sheet (attachment14)
      Merger ~44M items/s
      Tapeserver ~14MB/s
      Beam still down

14.11 beam back aida file R30_112

14.13 beam back down,  back at 14.16

14.20 no beam

14.23 notice some timestamp error in ucesb 37 minutes ago (about R30_102) no errors appeared on the MBS relay terminal,
      double by analysing R30_101,102,103 no timewarps
      beam is back but low intensity at the moment

14.46 beam looks to be back at a higher intensity now R30_127 (attachment15)

16.00 System wide checks all okay, no fails
      Statistics (attachment16)
      Rate spectra (attachment17)
      FEE temps (attament18)
      Leakage currents written to sheet (attachment19)
      Merger ~45M items/s
      Tapeserver ~14MB/s

18.00 System wide checks all okay, no fails
      Statistics (attachment20)
      Rate spectra (attachment21)
      FEE temps (attament22)
      Leakage currents written to sheet (attachment23)
      Merger ~45M items/s
      Tapeserver ~15MB/s

      bunch of timewarp errors in ucesb, no errors in MBS relay terminal though
      now see error in MBS relay!
      "Warning: At least one MIDAS block missed in relay" Warning was set up by NH and OH to detect if the LSB of the first ADC data item in a block is less than the LAST LSB of the previous data block. It then iterates the info code 4 and if needed 5 stored before resetting the LSB stored.
      It triggered 63 times.
      I have checked all the files in the vicinity and there are no timewarps in the MIDAS data visible to TD's analyser

19:10 MBS DAQ stopped to allow FRS tests

19:38 MBS DAQ is back

20:15 Aida07 fails FPGA check
			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
   
      All other system wide checks ok
      Statistics - attachment 24 (Again ignore the year in the filename)
      Temperatures - attachment 25
      Bias and leakage currents - 26

21:36 They are resetting the time sorter due to FATIMA issues

22:38 System wide checks error on WR in addition to the previous FPGA error
		 Base 		Current 	Difference
aida05 fault 	 0x5cc9 : 	 0x5cca : 	 1  
aida06 fault 	 0xde76 : 	 0xde77 : 	 1  
aida07 fault 	 0xa566 : 	 0xa567 : 	 1  
aida08 fault 	 0x6ab0 : 	 0x6ab1 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4
      Statistics - attachment 27
      Temperature - attachment 28
      Bias and leakage current - attachment 29

23:02 We still see timewarp errors in the ucesb. With our changes to the relay we can confirm that the WR headers in the MBS relay are time ordered. The issue must be downstream in the MBS chain.
Attachment 1: 210310_0805_Stats.png
210310_0805_Stats.png
Attachment 2: 210310_0806_Temp.png
210310_0806_Temp.png
Attachment 3: 210310_0806_Temp2.png
210310_0806_Temp2.png
Attachment 4: 210310_0808_Bias.png
210310_0808_Bias.png
Attachment 5: 210311_1039_Stats.png
210311_1039_Stats.png
Attachment 6: 210311_1039_Temp.png
210311_1039_Temp.png
Attachment 7: 210311_1041_Bias.png
210311_1041_Bias.png
Attachment 8: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 11: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 12: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 13: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 14: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 15: 1450_SpectraRate.PNG
1450_SpectraRate.PNG
Attachment 16: 1600_Statistics.PNG
1600_Statistics.PNG
Attachment 17: 1600_SpectraRate.PNG
1600_SpectraRate.PNG
Attachment 18: 1600_FEETemps.PNG
1600_FEETemps.PNG
Attachment 19: 1600_LeakageCurrents.PNG
1600_LeakageCurrents.PNG
Attachment 20: 1800_Statistics.PNG
1800_Statistics.PNG
Attachment 21: 1800_SpectraRate.PNG
1800_SpectraRate.PNG
Attachment 22: 1800_FEETemps.PNG
1800_FEETemps.PNG
Attachment 23: 1800_LeakageCurrents.PNG
1800_LeakageCurrents.PNG
Attachment 24: 200311_2014_Stats.png
200311_2014_Stats.png
Attachment 25: 200311_2014_Temp.png
200311_2014_Temp.png
Attachment 26: 210311_2017_Bias.png
210311_2017_Bias.png
Attachment 27: 210311_2225_sTATS.png
210311_2225_sTATS.png
Attachment 28: 210311_2226_Temp.png
210311_2226_Temp.png
Attachment 29: 210311_2227_Bias.png
210311_2227_Bias.png
  188   Thu Mar 11 00:38:33 2021 TDThusrday 11 March 00.00-08.00
01.38 DAQ continues OK - file R22_388

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      System wide checks all OK *except*

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.4M data items/s
      TapeServer OK - 14Mb/s

01.47 no merger server error/warning messages since last check by TD at start of shift 00.00

03.44 merger server error/warning messages since last check e.g.

MERGE Data Link (30260): bad timestamp  6 3 0xc1bd7e9a 0x0ce9c9c6 0x0000292e0ce9c9c6 0x166b292e0ce9c9c6 0x166b292e0cea5486

03.46 DAQ continues OK - file R22_441

      System wide checks all OK *except*

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

FPGA timestamp errors
			 Base 		Current 		Difference
aida07 fault 	 0x1 : 	 0x2 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.6M data items/s
      TapeServer OK - 15Mb/s

02.51 Rate spectra - attachment 7

04.14 p+n junction HEC spectra - attachment 8

      all spectra zero'd 

06.22 merger server error/warning messages since last check e.g.

MERGE Data Link (30265): bad timestamp  11 3 0xc2fc8067 0x02df391e 0x0000324e62df391e 0x166b324e62df391e 0x166b324e643c775e

06.23 DAQ continues OK - file R22_508

      System wide checks all OK *except*

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit decoder status
		 Base 		Current 	Difference
aida01 fault 	 0xe67 : 	 0xe68 : 	 1  
aida02 fault 	 0x6b22 : 	 0x6b23 : 	 1  
aida03 fault 	 0x4c76 : 	 0x4c77 : 	 1  
aida04 fault 	 0x7542 : 	 0x7543 : 	 1  
aida05 fault 	 0x1597 : 	 0x1599 : 	 2  
aida06 fault 	 0xe241 : 	 0xe243 : 	 2  
aida07 fault 	 0x749 : 	 0x74c : 	 3  
aida08 fault 	 0xf854 : 	 0xf856 : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp errors

			 Base 		Current 		Difference
aida07 fault 	 0x1 : 	 0x2 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      FEE64 Temperatures OK - attachment 9
      Good event statistics OK - attachment 10
      Detector bias & leakage currents OK - attachment 11
      Merger OK - 4.8M data items/s
      TapeServer OK - 14Mb/s
Attachment 1: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-11_Statistics_aidas-gsi.png
Screenshot_2021-03-11_Statistics_aidas-gsi.png
Attachment 3: 20.png
20.png
Attachment 4: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-11_Statistics_aidas-gsi(1).png
Screenshot_2021-03-11_Statistics_aidas-gsi(1).png
Attachment 6: 21.png
21.png
Attachment 7: Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-11_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-03-11_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 10: Screenshot_2021-03-11_Statistics_aidas-gsi(2).png
Screenshot_2021-03-11_Statistics_aidas-gsi(2).png
Attachment 11: 22.png
22.png
  187   Wed Mar 10 07:16:02 2021 OH, LSWednesday 10th March 08:00-24:00
08:16 System wide checks all ok
      Statistics - attachment 1
      Temperatures - attachment 2
      bias - attachment 3
      Merger running 4.5e6 events per second
      Tape data writing at 42MB per second

      Currently no beam

09:45 While there is no beam we will perform a longer test of the new merger.
      Run R20 stopped and merger changed to neew version
      Started R21.

      The now usual behaviour of additional files at start of merge observed.
      N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was 
set to going
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_0
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_1
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_2
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_3
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_4
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_5
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_6
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_7
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_8
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_9
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_10
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_11
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_12
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_13
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_14
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_15
-rw-rw-r--. 1 npg npg  64K Mar 10 09:50 R21_16
-rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17
-rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18

     TapeData rate of 13280 kB/sec
     No errors observed in ucesb
     Correlations seen between time machine in AIDA and Ge in online
     Pulser rate in the online makes sense
     With the current data rate remaining HDD space on current drive will last around 89 hours.

09.20 analysis of file R21_27
      ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data 


10:24 Message while performing system wide checks:
Get returned with an error
error: SOAP http transport timed out after 10000 ms
NONE
error: SOAP http transport timed out after 10000 ms
    while executing
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args"
    (procedure "XAIDAAccessClient__Get" line 1)
    invoked from within
"XAIDAAccessClient__Get $Addr"

     aida02 restarted itself during the reset process
     Looking at the log messages on aida02 cannot see any reason for the cause of the restart.
     System wide checks following the restart all ok.
     When restarting the MBS relay errors observed until a timestamp was observed:
     Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000

10:55 Statistics - attachment 5
      Temperature - attachment 6
      Bias - attachment 7

11:03 Time machine correlation spectra with new merger
      AIDA - FATIMA - Attachment 8
      AIDA - Ge - Attachment 9

11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the 
layout to determine which FEE/ASIC caused the events before checking ASIC control

12:12 System wide checks all ok *except adc calibration which is same as before
      Statistics - attachment 10
      Temperature - attachment 11
      Bias and leakage currents ok - attachment 12

      A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat.

      After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time.
      They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon 
cable from the VME scalers.
      Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of 
DSSD3.
    
12:50 We have waveforms for some FEES
      Layout 7 attachment -14
      Layout 8 - attachment 15

14.00 (LS)
      Still no beam
      System wide checks okay except same as before:
      **FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module**
      
      Statistics (attachment16)
      Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16):
      aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up
      aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on
 
      FEE Temps (attachment17)
      Leakage currents written to sheets(attachment18)
      Merger ~ 45M items/s
      TapeServer~ 14MB/s
     
14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file 
      R22_98, rate spectra attached (attachment19)

14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw 
      no timewarps so we are not losing anything.
      Restarted MBS relay, errors have not reappeared so far

16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
      Statistics (attachment20)
      Rate spectra (attachment21)
      FEE Temps (attachment22)
      Leakage currents written to sheets(attachment23)
      Merger ~ 4.3M items/s
      TapeServer~ 14MB/s  
      Current file R22_148   

16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from 
S452f115
      Around R22_150

18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm)
      Rates of R22_122 (attachment25):
      First DSSSD (fee0-fee3) ~66/s
      Second DSSSD (fee4-fee7) ~49/s
      Third DSSSD (fee8-fee11) ~22/s

      Rates of R22_178 (attachment26):
      First DSSSD (fee0-fee3) ~122/s
      Second DSSSD (fee4-fee7) ~94/s
      Third DSSSD (fee8-fee11) ~49/s


18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
      Statistics (attachment27)
      Rate spectra (attachment28)
      FEE Temps (attachment29)
      Leakage currents written to sheets(attachment30), look to be on the way down
      Merger ~ 4.5M items/s
      TapeServer~ 15MB/s  

18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier

19:34 MBS DAQ Crashed
20:00 DAQ is still down. They have Sultan working on it.

20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre 
reboot)
      We are borrowing a HDD from the SHIP group

20:30 System wide checks all ok
      Statistics - attachment 31
      Temperature - attachment 32
      Bias - attachment 33

22:10 MBS DAQ has crashed
      Merger terminal is now showing a large number of bad merge events

      It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS.
      

22:42 System wide checks all ok
      Statistics - attachment 34
      Temperature - Attachment 35
      Bias and leakage currents ok - attachment 36

23:33 System wide checks all ok - attachment 37
      Temperatures - attachment 38
      Bias and leakage currents ok - attachment 39
Attachment 1: 210310_0813_Stats.png
210310_0813_Stats.png
Attachment 2: 210310_0813_Temp.png
210310_0813_Temp.png
Attachment 3: 210310_0714_Bias.png
210310_0714_Bias.png
Attachment 4: R21_27
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259319480 ( 1687258.2 Hz)
        Other data format:    2600520 (   16920.2 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1272 (       8.3 Hz)
                               RESUME:       1272 (       8.3 Hz)
                              SYNC100:        573 (       3.7 Hz)
                              WR48-63:        573 (       3.7 Hz)
                           FEE64 disc:     302889 (    1970.7 Hz)
                             MBS info:    2293941 (   14925.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      153.693 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000              150.055
  2                0.129              153.277
  3                1.695              152.203
  4                0.000               97.174
  5               15.630              152.471
  6                0.533              122.407
  7                0.048              144.955
  8               17.221              151.934
  9               26.571              143.076
 10                0.063              148.982
 11               11.592              141.197
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   20482535       1736          0          0        381        381         52         52          0        870          0          0
  1   26566394       1030          0          0          0          0         56         56          0        918          0          0
  2   26980761    1455064          0          0          2          2         87         87     302889    1151997          0          0
  3   26884821    1140284          0          0          5          5         65         65          0    1140144          0          0
  4    7516935         23          0          0          0          0          7          7          0          9          0          0
  5   16098094       1219          0          0        577        577         31         31          0          3          0          0
  6    9441595         40          0          0          2          2         18         18          0          0          0          0
  7    8769482         38          0          0          2          2         17         17          0          0          0          0
  8   32618578        336          0          0         98         98         70         70          0          0          0          0
  9   32790396        426          0          0        136        136         77         77          0          0          0          0
 10   16087963         82          0          0          1          1         40         40          0          0          0          0
 11   35081926        242          0          0         68         68         53         53          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:37059.496s (    0.863 blocks/s,   0.054 Mb/s)
Attachment 5: 210310_1054_Stats.png
210310_1054_Stats.png
Attachment 6: 210310_1055_Temp.png
210310_1055_Temp.png
Attachment 7: 210310_1055_Bias.png
210310_1055_Bias.png
Attachment 8: AIDA_Fatima_Correlation.PNG
AIDA_Fatima_Correlation.PNG
Attachment 9: AIDA_Ge_Correlation.PNG
AIDA_Ge_Correlation.PNG
Attachment 10: 210310_1210_Stats.png
210310_1210_Stats.png
Attachment 11: 210310_1211_Temp.png
210310_1211_Temp.png
Attachment 12: 210310_1212_Bias.png
210310_1212_Bias.png
Attachment 13: AIDA_leakage.PNG
AIDA_leakage.PNG
Attachment 14: 210310_1348_Layout7.png
210310_1348_Layout7.png
Attachment 15: 210310_1252_Layout8.png
210310_1252_Layout8.png
Attachment 16: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 17: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 18: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 19: 1400_SpectraRate.PNG
1400_SpectraRate.PNG
Attachment 20: 1600_Statistics.PNG
1600_Statistics.PNG
Attachment 21: 1600_SpectraRate.PNG
1600_SpectraRate.PNG
Attachment 22: 1600_FEETemps.PNG
1600_FEETemps.PNG
Attachment 23: 1600_LeakageCurrents.PNG
1600_LeakageCurrents.PNG
Attachment 24: 1640_SpectraRate.PNG
1640_SpectraRate.PNG
Attachment 25: 1700_R22_122_Rates.PNG
1700_R22_122_Rates.PNG
Attachment 26: 1700_R22_178_Rates.PNG
1700_R22_178_Rates.PNG
Attachment 27: 1800_Statistics.PNG
1800_Statistics.PNG
Attachment 28: 1800_SpectraRate.PNG
1800_SpectraRate.PNG
Attachment 29: 1800_FEETemps.PNG
1800_FEETemps.PNG
Attachment 30: 1800_LeakageCurrents.PNG
1800_LeakageCurrents.PNG
Attachment 31: 210310_2039_Stats.png
210310_2039_Stats.png
Attachment 32: 210310_2040_Temp.png
210310_2040_Temp.png
Attachment 33: 210310_2041_Bias.png
210310_2041_Bias.png
Attachment 34: 210310_2241_Stats.png
210310_2241_Stats.png
Attachment 35: 210310_2241_Temp.png
210310_2241_Temp.png
Attachment 36: 210310_2242_Bias.png
210310_2242_Bias.png
Attachment 37: 210310_2331_Stats.png
210310_2331_Stats.png
Attachment 38: 210310_2332_Temp.png
210310_2332_Temp.png
Attachment 39: 210310_2333_Bias.png
210310_2333_Bias.png
  186   Tue Mar 9 23:12:17 2021 CAMarch 10th 00:00 - 08:00
00:13 beam back, but some spills 'missing'

      DESPEC keeping beam as is, but may stop later if it worsens

00:15 DAQ continues OK - file R20_622

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:20 System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2

      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1

     If any of these counts are reported as in error
     The ASIC readout system has detected a timeslip.
     That is the timestamp read from the time FIFO is not younger than the last

00:24 still no bad timestamp errors in NewMerger since 16:30 UTC

00:30 FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3

      Merger OK - 4.2M data items/s
      TapeServer OK - 45 Mb/s

01:30 rate spectra - attachment 4

      HEC spectra - attachment 5 & 6

      note aida04 spectrum still not showing

02:07 beam off - file R20_760

02:11 beam back - file R20_764

02:44 System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2


      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1


      still no bad timestamp errors in NewMerger since 16:30 UTC

      FEE64 Temperatures OK - attachment 7
      Good event statistics OK - attachment 8
      Detector bias & leakage currents OK - attachment 9

      Merger OK - 4.2M data items/s
      TapeServer OK - 44 Mb/s

04:16 bad timestamp errors in NewMerger terminal, first since 16:30 UTC - attachment 10

      System wide checks all OK *except*

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2


      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1


      FEE64 Temperatures OK - attachment 11
      Good event statistics OK - attachment 12
      Detector bias & leakage currents OK - attachment 13

05:03 Merger OK - 4.5M data items/s
      TapeServer OK - 43 Mb/s

      no bad timestamp errors for ~40 mins, data forwarding to MBS ok at usual rate

06:11 rates spectra - attachment 14

06:14 System wide checks

      ADC Calibration
      FEE64 module aida07 failed
      FEE64 module aida10 failed
      Calibration test result: Passed 10, Failed 2

      Check WR decoder status
      Base 		Current 	Difference
      aida05 fault 	 0x1591 : 	 0x1593 : 	 2  
      aida06 fault 	 0xe23c : 	 0xe23e : 	 2  
      aida07 fault 	 0x743 : 	 0x745 : 	 2  
      aida08 fault 	 0xf84e : 	 0xf850 : 	 2  
      aida09 fault 	 0x5449 : 	 0x544a : 	 1  
      White Rabbit error counter test result: Passed 7, Failed 5

      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
   

      Check FPGA Timestamp Errors
      Base 		Current 		Difference
      aida12 fault 	 0x0 : 	 0x1 : 	 1  
      FPGA Timestamp error counter test result: Passed 11, Failed 1

      still no further bad timestamp errors in NewMerger terminal

      collected all WR and FPGA errors from baseline, system wide checks ok

      FEE64 Temperatures OK - attachment 15
      Good event statistics OK - attachment 16
      Detector bias & leakage currents OK - attachment 17

06:24 AIDA writing to file R20_1080

      Merger OK - 4.5M data items/s
      TapeServer OK - 46 Mb/s

06:26 another burst of bad timestamp errors - attachment 18

      performed system wide checks again - all ok aside from aida07/aida10 calibration

07:32 HEC spectra - attachments 19/20

07:35 no beam

      


      
Attachment 1: Screenshot_from_2021-03-09_23-27-20.png
Screenshot_from_2021-03-09_23-27-20.png
Attachment 2: Screenshot_from_2021-03-09_23-29-22.png
Screenshot_from_2021-03-09_23-29-22.png
Attachment 3: Screenshot_from_2021-03-09_23-29-39.png
Screenshot_from_2021-03-09_23-29-39.png
Attachment 4: Screenshot_from_2021-03-10_00-27-45.png
Screenshot_from_2021-03-10_00-27-45.png
Attachment 5: Screenshot_from_2021-03-10_00-28-39.png
Screenshot_from_2021-03-10_00-28-39.png
Attachment 6: Screenshot_from_2021-03-10_00-29-25.png
Screenshot_from_2021-03-10_00-29-25.png
Attachment 7: Screenshot_from_2021-03-10_01-40-24.png
Screenshot_from_2021-03-10_01-40-24.png
Attachment 8: Screenshot_from_2021-03-10_01-42-57.png
Screenshot_from_2021-03-10_01-42-57.png
Attachment 9: Screenshot_from_2021-03-10_01-43-39_-_1.png
Screenshot_from_2021-03-10_01-43-39_-_1.png
Attachment 10: Screenshot_from_2021-03-10_03-17-56.png
Screenshot_from_2021-03-10_03-17-56.png
Attachment 11: Screenshot_from_2021-03-10_03-23-03.png
Screenshot_from_2021-03-10_03-23-03.png
Attachment 12: Screenshot_from_2021-03-10_03-23-26.png
Screenshot_from_2021-03-10_03-23-26.png
Attachment 13: Screenshot_from_2021-03-10_03-23-43.png
Screenshot_from_2021-03-10_03-23-43.png
Attachment 14: Screenshot_from_2021-03-10_05-10-19.png
Screenshot_from_2021-03-10_05-10-19.png
Attachment 15: Screenshot_from_2021-03-10_05-17-39.png
Screenshot_from_2021-03-10_05-17-39.png
Attachment 16: Screenshot_from_2021-03-10_05-18-04.png
Screenshot_from_2021-03-10_05-18-04.png
Attachment 17: Screenshot_from_2021-03-10_05-21-47.png
Screenshot_from_2021-03-10_05-21-47.png
Attachment 18: Screenshot_from_2021-03-10_05-27-03.png
Screenshot_from_2021-03-10_05-27-03.png
Attachment 19: Screenshot_from_2021-03-10_06-31-51.png
Screenshot_from_2021-03-10_06-31-51.png
Attachment 20: Screenshot_from_2021-03-10_06-32-18.png
Screenshot_from_2021-03-10_06-32-18.png
  185   Tue Mar 9 20:04:20 2021 TDTuesday 9 March
21.04 DAQ continues OK - file R20_379

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      System wide checks all OK *except*

ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

Check FPGA Timestamp Errors
Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1

If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.2M data items/s
      TapeServer OK - 45 Mb/s

21.20 no merger server error/warning messages since 16.30UTC

21.35 p+n junction strip  HEC & LEC spectra, Rate spectra common y-scale 0-30000 & 0-10
      all spectra zero'd

22.05 no beam for 2-3h whilst accelerator generator is fixed
Attachment 1: Screenshot_2021-03-09_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-09_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-09_Statistics_aidas-gsi.png
Screenshot_2021-03-09_Statistics_aidas-gsi.png
Attachment 3: 10.png
10.png
Attachment 4: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-09_Spectrum_Browser_aidas-gsi(3).png
  184   Tue Mar 9 10:23:45 2021 TDAnalysis of R14 (w/ NewMerger with min info code 4 & 5 data items)
Per https://elog.ph.ed.ac.uk/DESPEC/183 R14 data files have been analysed

I think we still need  paired info code 4 & 5 data items.
I also don't understand the pattern of file sizes ... 'old' TapeServer buffers being flushed?

[npg@aidas-gsi analyser]$ ls -lrth /TapeData/S452/R14*
-rw-rw-r--. 1 npg npg  64K Mar  9 10:07 /TapeData/S452/R14_0
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_1
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_2
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_3
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_4
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_5
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_6
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_7
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_8
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_9
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_10
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_11
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_12
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_13
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_14
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_15
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_16
-rw-rw-r--. 1 npg npg  64K Mar  9 10:23 /TapeData/S452/R14_17
-rw-rw-r--. 1 npg npg 363M Mar  9 10:23 /TapeData/S452/R14_18
-rw-rw-r--. 1 npg npg 2.0G Mar  9 10:26 /TapeData/S452/R14_19
-rw-rw-r--. 1 npg npg 1.6G Mar  9 10:28 /TapeData/S452/R14_20

R14_0 (attachment 1) has multiple PAUSE/RESUME data items at start of file (R14_0) followed by 1x info code 4 & 5 data items (as expected)

Also true for other short (64k) files - multiple PAUSE/RESUME data items, 1x info code 4 & 5 data items, and ADC data from one FEE64 module only

[npg@aidas-gsi analyser]$ ./analyser v /TapeData/S452/R14_0 | more
 *** TDR format 3.3.0 analyser - TD - January 2019
  verbose
 ***    RESUME timestamp: block:     1 ptr:     7 data: 0x843A160B module:  4 information type:  3 information field: 0x000A160B ts: 0x0000A160B1733D66
 ***     PAUSE timestamp: block:     1 ptr:     9 data: 0x842A160B module:  4 information type:  2 information field: 0x000A160B ts: 0x0000A160B6508028
 ***    RESUME timestamp: block:     1 ptr:    11 data: 0x873A160B module:  7 information type:  3 information field: 0x000A160B ts: 0x0000A160B7727448
 ***    RESUME timestamp: block:     1 ptr:    13 data: 0x823A160B module:  2 information type:  3 information field: 0x000A160B ts: 0x0000A160B775CA9E
 ***     PAUSE timestamp: block:     1 ptr:    15 data: 0x872A160B module:  7 information type:  2 information field: 0x000A160B ts: 0x0000A160BC4F8028
 ***     PAUSE timestamp: block:     1 ptr:    17 data: 0x822A160B module:  2 information type:  2 information field: 0x000A160B ts: 0x0000A160BC548028
 ***    RESUME timestamp: block:     1 ptr:    19 data: 0x863A160B module:  6 information type:  3 information field: 0x000A160B ts: 0x0000A160BFE804B2
 ***     PAUSE timestamp: block:     1 ptr:    21 data: 0x862A160C module:  6 information type:  2 information field: 0x000A160C ts: 0x0000A160C4C70028
 ***    RESUME timestamp: block:     1 ptr:    23 data: 0x803A160C module:  0 information type:  3 information field: 0x000A160C ts: 0x0000A160C863646A
 ***    RESUME timestamp: block:     1 ptr:    25 data: 0x833A160C module:  3 information type:  3 information field: 0x000A160C ts: 0x0000A160CC641992
 ***     PAUSE timestamp: block:     1 ptr:    27 data: 0x802A160C module:  0 information type:  2 information field: 0x000A160C ts: 0x0000A160CD410028
 ***     PAUSE timestamp: block:     1 ptr:    29 data: 0x832A160D module:  3 information type:  2 information field: 0x000A160D ts: 0x0000A160D1420028
 ***    RESUME timestamp: block:     1 ptr:    31 data: 0x853A160D module:  5 information type:  3 information field: 0x000A160D ts: 0x0000A160D3D39E8C
 ***    RESUME timestamp: block:     1 ptr:    33 data: 0x883A160D module:  8 information type:  3 information field: 0x000A160D ts: 0x0000A160D3D4D5EA
 ***     PAUSE timestamp: block:     1 ptr:    35 data: 0x852A160D module:  5 information type:  2 information field: 0x000A160D ts: 0x0000A160D8B30028
 ***     PAUSE timestamp: block:     1 ptr:    37 data: 0x882A160D module:  8 information type:  2 information field: 0x000A160D ts: 0x0000A160D8B30028
 ***    RESUME timestamp: block:     1 ptr:    39 data: 0x813A160D module:  1 information type:  3 information field: 0x000A160D ts: 0x0000A160D9F76CF8
 ***     PAUSE timestamp: block:     1 ptr:    41 data: 0x812A160D module:  1 information type:  2 information field: 0x000A160D ts: 0x0000A160DED50028
 ***    RESUME timestamp: block:     1 ptr:    43 data: 0x843A160F module:  4 information type:  3 information field: 0x000A160F ts: 0x0000A160F7AD970E
 ***    RESUME timestamp: block:     1 ptr:    45 data: 0x8A3A160F module: 10 information type:  3 information field: 0x000A160F ts: 0x0000A160F938391C
 ***    RESUME timestamp: block:     1 ptr:    47 data: 0x893A160F module:  9 information type:  3 information field: 0x000A160F ts: 0x0000A160FDC0600E
 ***     PAUSE timestamp: block:     1 ptr:    49 data: 0x8A2A160F module: 10 information type:  2 information field: 0x000A160F ts: 0x0000A160FE178028
 ***    RESUME timestamp: block:     1 ptr:    51 data: 0x823A1610 module:  2 information type:  3 information field: 0x000A1610 ts: 0x0000A161009DF9B2
 ***    RESUME timestamp: block:     1 ptr:    53 data: 0x873A1610 module:  7 information type:  3 information field: 0x000A1610 ts: 0x0000A16100B7FD08
 ***     PAUSE timestamp: block:     1 ptr:    55 data: 0x892A1610 module:  9 information type:  2 information field: 0x000A1610 ts: 0x0000A161029F8028
 ***    RESUME timestamp: block:     1 ptr:    57 data: 0x863A1610 module:  6 information type:  3 information field: 0x000A1610 ts: 0x0000A16108134396
 ***    RESUME timestamp: block:     1 ptr:    59 data: 0x803A1610 module:  0 information type:  3 information field: 0x000A1610 ts: 0x0000A1610C484524
 ***    RESUME timestamp: block:     1 ptr:    61 data: 0x833A1610 module:  3 information type:  3 information field: 0x000A1610 ts: 0x0000A1610F8B0910
 ***    RESUME timestamp: block:     1 ptr:    63 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16112BA0000
 ***    RESUME timestamp: block:     1 ptr:    65 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16112E48000
 ***    RESUME timestamp: block:     1 ptr:    67 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161130F0000
 ***    RESUME timestamp: block:     1 ptr:    69 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16113398000
 ***    RESUME timestamp: block:     1 ptr:    71 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16113640000
 ***    RESUME timestamp: block:     1 ptr:    73 data: 0x883A1611 module:  8 information type:  3 information field: 0x000A1611 ts: 0x0000A161137B1560
 ***    RESUME timestamp: block:     1 ptr:    75 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161138E8000
 ***    RESUME timestamp: block:     1 ptr:    77 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16113B90000
 ***    RESUME timestamp: block:     1 ptr:    79 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16113E38000
 ***    RESUME timestamp: block:     1 ptr:    81 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161140E0000
 ***    RESUME timestamp: block:     1 ptr:    83 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16114388000
 ***    RESUME timestamp: block:     1 ptr:    85 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16114630000
 ***    RESUME timestamp: block:     1 ptr:    87 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161148D8000
 ***    RESUME timestamp: block:     1 ptr:    89 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16114B80000
 ***    RESUME timestamp: block:     1 ptr:    91 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16114E28000
 ***    RESUME timestamp: block:     1 ptr:    93 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161150D0000
 ***    RESUME timestamp: block:     1 ptr:    95 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16115378000
 ***    RESUME timestamp: block:     1 ptr:    97 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16115620000
 ***    RESUME timestamp: block:     1 ptr:    99 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161158C8000
 ***    RESUME timestamp: block:     1 ptr:   101 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16115B70000
 ***    RESUME timestamp: block:     1 ptr:   103 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16115E18000
 ***    RESUME timestamp: block:     1 ptr:   105 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161160C0000
 ***    RESUME timestamp: block:     1 ptr:   107 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16116368000
 ***    RESUME timestamp: block:     1 ptr:   109 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16116610000
 ***    RESUME timestamp: block:     1 ptr:   111 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161168B8000
 ***    RESUME timestamp: block:     1 ptr:   113 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16116B60000
 ***    RESUME timestamp: block:     1 ptr:   115 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16116E08000
 ***    RESUME timestamp: block:     1 ptr:   117 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161170B0000
 ***    RESUME timestamp: block:     1 ptr:   119 data: 0x853A1611 module:  5 information type:  3 information field: 0x000A1611 ts: 0x0000A161171BADC4
 ***    RESUME timestamp: block:     1 ptr:   121 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16117358000
 ***    RESUME timestamp: block:     1 ptr:   123 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A16117600000
 ***    RESUME timestamp: block:     1 ptr:   125 data: 0x8B3A1611 module: 11 information type:  3 information field: 0x000A1611 ts: 0x0000A161178A8000
 ***    RESUME timestamp: block:     1 ptr:   127 data: 0x813A1611 module:  1 information type:  3 information field: 0x000A1611 ts: 0x0000A1611A161F3C
 ***    RESUME timestamp: block:     1 ptr:   129 data: 0x8A3A1613 module: 10 information type:  3 information field: 0x000A1613 ts: 0x0000A1613C6AAE72
 ***    RESUME timestamp: block:     1 ptr:   131 data: 0x893A1613 module:  9 information type:  3 information field: 0x000A1613 ts: 0x0000A1613F230BBE
 ***   WR48-64 timestamp: block:     1 ptr:   133 data: 0x8450166A module:  4 information type:  5 information field: 0x0000166A ts: 0x166AA1613483FE84
 ***   SYNC100 timestamp: block:     1 ptr:   135 data: 0x844A1622 module:  4 information type:  4 information field: 0x000A1622 ts: 0x166AA1622483FE84
 ***            ADC data: block:     1 ptr:   137 data: 0xC13C7FEB module:  4 fail: 0 range: 0 id:   316 channel: 60 adc: 32747  ts: 0x166AA1622483FE84
 ***            ADC data: block:     1 ptr:   139 data: 0xC1067E53 module:  4 fail: 0 range: 0 id:   262 channel:  6 adc: 32339  ts: 0x166AA16224840654
 ***            ADC data: block:     1 ptr:   141 data: 0xC1167E4C module:  4 fail: 0 range: 0 id:   278 channel: 22 adc: 32332  ts: 0x166AA16224840654
 ***            ADC data: block:     1 ptr:   143 data: 0xC1247EAF module:  4 fail: 0 range: 0 id:   292 channel: 36 adc: 32431  ts: 0x166AA16224840654
 ***            ADC data: block:     1 ptr:   145 data: 0xC11F7D3F module:  4 fail: 0 range: 0 id:   287 channel: 31 adc: 32063  ts: 0x166AA16224840E24
 ***            ADC data: block:     1 ptr:   147 data: 0xC12D7EF0 module:  4 fail: 0 range: 0 id:   301 channel: 45 adc: 32496  ts: 0x166AA16224840E24
 ***            ADC data: block:     1 ptr:   149 data: 0xC1117E24 module:  4 fail: 0 range: 0 id:   273 channel: 17 adc: 32292  ts: 0x166AA16224842594
 ***            ADC data: block:     1 ptr:   151 data: 0xC11E7D06 module:  4 fail: 0 range: 0 id:   286 channel: 30 adc: 32006  ts: 0x166AA16224844CA4
 ***            ADC data: block:     1 ptr:   153 data: 0xC1217EB3 module:  4 fail: 0 range: 0 id:   289 channel: 33 adc: 32435  ts: 0x166AA16224845474
 ***            ADC data: block:     1 ptr:   155 data: 0xC1347E05 module:  4 fail: 0 range: 0 id:   308 channel: 52 adc: 32261  ts: 0x166AA16224845474
 ***            ADC data: block:     1 ptr:   157 data: 0xC1037EBC module:  4 fail: 0 range: 0 id:   259 channel:  3 adc: 32444  ts: 0x166AA16224845C44
 ***            ADC data: block:     1 ptr:   159 data: 0xC13E7E99 module:  4 fail: 0 range: 0 id:   318 channel: 62 adc: 32409  ts: 0x166AA16224845C44
 ***            ADC data: block:     1 ptr:   161 data: 0xC1047D6F module:  4 fail: 0 range: 0 id:   260 channel:  4 adc: 32111  ts: 0x166AA16224846414
 ***            ADC data: block:     1 ptr:   163 data: 0xC1137EE5 module:  4 fail: 0 range: 0 id:   275 channel: 19 adc: 32485  ts: 0x166AA16224846414
 ***            ADC data: block:     1 ptr:   165 data: 0xC10E7F30 module:  4 fail: 0 range: 0 id:   270 channel: 14 adc: 32560  ts: 0x166AA16224846BE4
 ***            ADC data: block:     1 ptr:   167 data: 0xC1157D12 module:  4 fail: 0 range: 0 id:   277 channel: 21 adc: 32018  ts: 0x166AA16224846BE4
 ***            ADC data: block:     1 ptr:   169 data: 0xC1287DD3 module:  4 fail: 0 range: 0 id:   296 channel: 40 adc: 32211  ts: 0x166AA16224846BE4
 ***            ADC data: block:     1 ptr:   171 data: 0xC1197E69 module:  4 fail: 0 range: 0 id:   281 channel: 25 adc: 32361  ts: 0x166AA162248473B4
 ***            ADC data: block:     1 ptr:   173 data: 0xC11B7EC0 module:  4 fail: 0 range: 0 id:   283 channel: 27 adc: 32448  ts: 0x166AA16224847B84
 ***            ADC data: block:     1 ptr:   175 data: 0xC1017D82 module:  4 fail: 0 range: 0 id:   257 channel:  1 adc: 32130  ts: 0x166AA16224848354
 ***            ADC data: block:     1 ptr:   177 data: 0xC1247E94 module:  4 fail: 0 range: 0 id:   292 channel: 36 adc: 32404  ts: 0x166AA162248492F4
 ***            ADC data: block:     1 ptr:   179 data: 0xC1047D33 module:  4 fail: 0 range: 0 id:   260 channel:  4 adc: 32051  ts: 0x166AA1622484F884
 ***            ADC data: block:     1 ptr:   181 data: 0xC13C7FFC module:  4 fail: 0 range: 0 id:   316 channel: 60 adc: 32764  ts: 0x166AA16224853ED4
 ***            ADC data: block:     1 ptr:   183 data: 0xC1067E6A module:  4 fail: 0 range: 0 id:   262 channel:  6 adc: 32362  ts: 0x166AA16224855E14
 ***            ADC data: block:     1 ptr:   185 data: 0xC1117E44 module:  4 fail: 0 range: 0 id:   273 channel: 17 adc: 32324  ts: 0x166AA16224858CF4
 ***            ADC data: block:     1 ptr:   187 data: 0xC1167EA2 module:  4 fail: 0 range: 0 id:   278 channel: 22 adc: 32418  ts: 0x166AA162248594C4
 ***            ADC data: block:     1 ptr:   189 data: 0xC1017EFB module:  4 fail: 0 range: 0 id:   257 channel:  1 adc: 32507  ts: 0x166AA1622485B404
 ***            ADC data: block:     1 ptr:   191 data: 0xC1017F83 module:  4 fail: 0 range: 0 id:   257 channel:  1 adc: 32643  ts: 0x166AA16224864874
 ***            ADC data: block:     1 ptr:   193 data: 0xC1197E3C module:  4 fail: 0 range: 0 id:   281 channel: 25 adc: 32316  ts: 0x166AA162248667B4
 ***            ADC data: block:     1 ptr:   195 data: 0xC1017DF9 module:  4 fail: 0 range: 0 id:   257 channel:  1 adc: 32249  ts: 0x166AA1622486DCE4
 ***            ADC data: block:     1 ptr:   197 data: 0xC1117E4E module:  4 fail: 0 range: 0 id:   273 channel: 17 adc: 32334  ts: 0x166AA162248759E4
 ***            ADC data: block:     1 ptr:   199 data: 0xC1167E55 module:  4 fail: 0 range: 0 id:   278 channel: 22 adc: 32341  ts: 0x166AA1622487AFD4
 ***            ADC data: block:     1 ptr:   201 data: 0xC1247E58 module:  4 fail: 0 range: 0 id:   292 channel: 36 adc: 32344  ts: 0x166AA1622487B7A4
 ***            ADC data: block:     1 ptr:   203 data: 0xC10B7DFE module:  4 fail: 0 range: 0 id:   267 channel: 11 adc: 32254  ts: 0x166AA1622487CF14
 ***            ADC data: block:     1 ptr:   205 data: 0xC12B7E6F module:  4 fail: 0 range: 0 id:   299 channel: 43 adc: 32367  ts: 0x166AA1622487CF14
 ***            ADC data: block:     1 ptr:   207 data: 0xC1157D3B module:  4 fail: 0 range: 0 id:   277 channel: 21 adc: 32059  ts: 0x166AA1622487D6E4
 ***            ADC data: block:     1 ptr:   209 data: 0xC1347E96 module:  4 fail: 0 range: 0 id:   308 channel: 52 adc: 32406  ts: 0x166AA1622487D6E4
 ***            ADC data: block:     1 ptr:   211 data: 0xC11A7D38 module:  4 fail: 0 range: 0 id:   282 channel: 26 adc: 32056  ts: 0x166AA1622487DEB4
 ***            ADC data: block:     1 ptr:   213 data: 0xC13C7F8E module:  4 fail: 0 range: 0 id:   316 channel: 60 adc: 32654  ts: 0x166AA1622487DEB4
 ***            ADC data: block:     1 ptr:   215 data: 0xC11B7E79 module:  4 fail: 0 range: 0 id:   283 channel: 27 adc: 32377  ts: 0x166AA1622487E684
 ***            ADC data: block:     1 ptr:   217 data: 0xC13D7EDB module:  4 fail: 0 range: 0 id:   317 channel: 61 adc: 32475  ts: 0x166AA1622487E684
 ***            ADC data: block:     1 ptr:   219 data: 0xC1347EB8 module:  4 fail: 0 range: 0 id:   308 channel: 52 adc: 32440  ts: 0x166AA16224886B54
 ***            ADC data: block:     1 ptr:   221 data: 0xC13C7F67 module:  4 fail: 0 range: 0 id:   316 channel: 60 adc: 32615  ts: 0x166AA16224887324
 ***            ADC data: block:     1 ptr:   223 data: 0xC13D7F18 module:  4 fail: 0 range: 0 id:   317 channel: 61 adc: 32536  ts: 0x166AA1622488F7F4
 ***            ADC data: block:     1 ptr:   225 data: 0xC1157D5C module:  4 fail: 0 range: 0 id:   277 channel: 21 adc: 32092  ts: 0x166AA16224890794
 ***            ADC data: block:     1 ptr:   227 data: 0xC1047DED module:  4 fail: 0 range: 0 id:   260 channel:  4 adc: 32237  ts: 0x166AA16224891F04
 ***            ADC data: block:     1 ptr:   229 data: 0xC1127E66 module:  4 fail: 0 range: 0 id:   274 channel: 18 adc: 32358  ts: 0x166AA16224891F04
--More--

R14_19 (attachment 2) looks OK - ignore rate information as this is based upon incomplete timestamp data
Attachment 1: R14_0
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:          1
          ADC data format:       8120 (   96053.8 Hz)
        Other data format:         65 (     768.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         11 (     130.1 Hz)
                               RESUME:         52 (     615.1 Hz)
                              SYNC100:          1 (      11.8 Hz)
                              WR48-63:          1 (      11.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:        0.085 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.994                0.000
  2                1.146                0.000
  3                1.045                0.000
  4                1.097                0.000
  5                1.047                0.000
  6                1.129                0.000
  7                1.148                0.000
  8                0.986                0.000
  9                1.015                0.000
 10                1.046                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0          0          3          0          0          1          2          0          0          0          0          0          0
  1          0          3          0          0          1          2          0          0          0          0          0          0
  2          0          3          0          0          1          2          0          0          0          0          0          0
  3          0          3          0          0          1          2          0          0          0          0          0          0
  4       8120          5          0          0          1          2          1          1          0          0          0          0
  5          0          3          0          0          1          2          0          0          0          0          0          0
  6          0          3          0          0          1          2          0          0          0          0          0          0
  7          0          3          0          0          1          2          0          0          0          0          0          0
  8          0          3          0          0          1          2          0          0          0          0          0          0
  9          0          3          0          0          1          2          0          0          0          0          0          0
 10          0          3          0          0          1          2          0          0          0          0          0          0
 11          0         30          0          0          0         30          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:    0.004s (  256.000 blocks/s,  16.000 Mb/s)
Attachment 2: R14_19
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  261598852 ( 1789659.8 Hz)
        Other data format:     321148 (    2197.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       1009 (       6.9 Hz)
                               RESUME:       1010 (       6.9 Hz)
                              SYNC100:        545 (       3.7 Hz)
                              WR48-63:        545 (       3.7 Hz)
                           FEE64 disc:     317229 (    2170.2 Hz)
                             MBS info:        810 (       5.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      146.172 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.992              139.586
  2                0.853              142.002
  3                1.182              142.271
  4                0.575              139.050
  5                6.293              131.265
  6                0.208              139.586
  7                0.037              142.002
  8               39.955              138.244
  9                9.898              143.613
 10                0.448              141.465
 11               32.263              145.760
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   21196377       1535          0          0        326        327         36         36          0        810          0          0
  1   25527261        100          0          0          4          4         46         46          0          0          0          0
  2   26242581     317335          0          0          6          6         47         47     317229          0          0          0
  3   27512430        132          0          0          5          5         61         61          0          0          0          0
  4    8193786         46          0          0          4          4         19         19          0          0          0          0
  5   17476404        516          0          0        220        220         38         38          0          0          0          0
  6   12155075         60          0          0          5          5         25         25          0          0          0          0
  7    9702993         52          0          0          1          1         25         25          0          0          0          0
  8   33325064        534          0          0        206        206         61         61          0          0          0          0
  9   34533675        282          0          0         56         56         85         85          0          0          0          0
 10   12187954         94          0          0          2          2         45         45          0          0          0          0
 11   33545252        462          0          0        174        174         57         57          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:40890.051s (    0.783 blocks/s,   0.049 Mb/s)
  183   Tue Mar 9 06:58:14 2021 CA, LSMarch 9th 08:00-11:00 shift
08:00 DAQ continues ok - writing to file R13_120

      Merger ok - 4.6M data items/s

      TapeServer ok - 43 MB/s

08:03 - all system wide checks ok

      - Temperatures ok - attachment 1

      - good event statistics ok - attachment 2

      - detector bias/leakage currents ok - attachment 3

08:08 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

08:32 CA briefly lost network connection - back now

08:42 DESPEC on file 102

09:24 Rate spectra - attachment 4

09:32 HEC spectra - attachment 5/6

      beam off - writing to file R13_243

09:42 merger bad timestamp errors - attachment 7

      all system wide checks ok

10:30 NH - Beam was down, I try the new merger from Vic again (R14)
           I made a mistake and had to powercycle fees as they got stuck! aida10 seemed to reboot itslef
           After fixing my mistakes, all running with new merger and rate 15 MB/s (R14)
           It all looked OK even going to MBS/Go4
           I reverted back to original hot merger, R15 and look at the data.
           Beam is now back and I return AIDA to shifters :)

11.30  System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Reran calibration for both FEEs  but still fail
       Statistics (attachment8)
       Rate spectra (attachment9)
       FEE temps (attachment10)
       Leakage currents written to sheet (attachment11)
       Tapeserver ~33MB/s
       Merger ~ 3.4M items/s
       AIDA file R15_92

11.55 Beam off for a couple minutes but back now

12.00 New Merge terminal disappeared so stopped daq and restarted, and also restarted MBS back working
      Now writing to R16
      Merger ~ 4.5M items/s
      Tapeserver ~44MB/s

12.41 no beam

13.00 DAQ crashed (aida03 no responding), AIDA recovered from power cycle while beam is down
      Beam is back now, forwarding to MBS
      Merger ~ 45M items/s
      Tapeserver ~44MB/s
      Writing to R17

13.20 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment12)
       Rate spectra (attachment13)
       FEE temps (attachment14)
       Leakage currents written to sheet (attachment15)
       Tapeserver ~45MB/s
       Merger ~ 4.4M items/s

14.27 beam off
      file R17_115
      back a couple minutes later

14.40 no beam again back at 14.46

15.40 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment16)
       Rate spectra (attachment17)
       FEE temps (attachment18)
       Leakage currents written to sheet (attachment19)
       Tapeserver ~44MB/s
       Merger ~ 4.4M items/s
       
15.52 beam down R17_221, back at 15.58 R17_229

16.07 Beam down
       
16:20 Removed S480 files from /media/1e... and /media/SecondDrive
      Set merger to no output and stopped tapedata.
      Changed symbolic link of /TapeData to /media/1e.....
      Restarted tapedata and the merger with outputting to file R20
      Local DESPEC was made aware of this before carrying out and stopped the run.
      Beam back

17.12 beam down for a couple of minutes R20_90

17.40 System wide checks okay except ADC calibration:
       
       FEE64 module aida07 failed
       FEE64 module aida10 failed
       Calibration test result: Passed 10, Failed 2
       If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
       calibration for that module

       Statistics (attachment20)
       Rate spectra (attachment21)
       FEE temps (attachment22)
       Leakage currents written to sheet (attachment23)
       Tapeserver ~45MB/s
       Merger ~ 4.5M items/s

      
Attachment 1: Screenshot_from_2021-03-09_07-05-35.png
Screenshot_from_2021-03-09_07-05-35.png
Attachment 2: Screenshot_from_2021-03-09_07-06-17.png
Screenshot_from_2021-03-09_07-06-17.png
Attachment 3: Screenshot_from_2021-03-09_07-06-53.png
Screenshot_from_2021-03-09_07-06-53.png
Attachment 4: Screenshot_from_2021-03-09_08-08-45_-_1.png
Screenshot_from_2021-03-09_08-08-45_-_1.png
Attachment 5: Screenshot_from_2021-03-09_08-24-06.png
Screenshot_from_2021-03-09_08-24-06.png
Attachment 6: Screenshot_from_2021-03-09_08-26-47.png
Screenshot_from_2021-03-09_08-26-47.png
Attachment 7: Screenshot_from_2021-03-09_08-44-17.png
Screenshot_from_2021-03-09_08-44-17.png
Attachment 8: 1130_Statistics.PNG
1130_Statistics.PNG
Attachment 9: 1130_SpectraRates.PNG
1130_SpectraRates.PNG
Attachment 10: 1130_FEETemps.PNG
1130_FEETemps.PNG
Attachment 11: 1130_LeakageCurrents.PNG
1130_LeakageCurrents.PNG
Attachment 12: 1330_Statistics.PNG
1330_Statistics.PNG
Attachment 13: 1330_SpectraRates.PNG
1330_SpectraRates.PNG
Attachment 14: 1330_FEETemps.PNG
1330_FEETemps.PNG
Attachment 15: 1330_LeakageCurrents.PNG
1330_LeakageCurrents.PNG
Attachment 16: 1540_Statistics.PNG
1540_Statistics.PNG
Attachment 17: 1540_SpectraRates.PNG
1540_SpectraRates.PNG
Attachment 18: 1540_FEETemps.PNG
1540_FEETemps.PNG
Attachment 19: 1540_LeakageCurrents.PNG
1540_LeakageCurrents.PNG
Attachment 20: 1730_Statistics.PNG
1730_Statistics.PNG
Attachment 21: 1730_SpectraRates.PNG
1730_SpectraRates.PNG
Attachment 22: 1730_FEETemps.PNG
1730_FEETemps.PNG
Attachment 23: 1730_LeakageCurrents.PNG
1730_LeakageCurrents.PNG
  182   Mon Mar 8 22:51:47 2021 OHTuesday 9th March
18:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

00:12 Noticed this log in UCESB.
Closed connection [140.181.60.97]...
1 clients...
AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848)
AIDA timewarp is over, skipped 2728 AIDA event(s)
=> Not emitting timewarped event (before 166a7a92fdac5c02)
Recovered from timewarp but skipped 5 event(s)

at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961
          Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either
          Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message

01:09 Another timewarp message
AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974)
=> Not emitting timewarped event (before 166a83bc6226e9ea)
Recovered from timewarp but skipped 6 event(s)
AIDA timewarp is over, skipped 323841 AIDA event

Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175
Again no timewarps in analysis of file.


02:00 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 4
      FEE Temperatures - attachment 5
      Bias and leakage currents ok - attachment 6

03:10 Problem with accelerator will be no beam for ~30 minutes


03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place.
      There are now also errors in all FEEs for WR events
	
		 Base 		Current 	Difference
aida01 fault 	 0xc45f : 	 0xc46a : 	 11  
aida02 fault 	 0xd250 : 	 0xd25b : 	 11  
aida03 fault 	 0x27b2 : 	 0x27bd : 	 11  
aida04 fault 	 0x714b : 	 0x7156 : 	 11  
aida05 fault 	 0x74f5 : 	 0x7500 : 	 11  
aida06 fault 	 0x558a : 	 0x5595 : 	 11  
aida07 fault 	 0x8d20 : 	 0x8d2b : 	 11  
aida08 fault 	 0x8131 : 	 0x813c : 	 11  
aida09 fault 	 0xf412 : 	 0xf41d : 	 11  
aida10 fault 	 0x82c2 : 	 0x82cd : 	 11  
aida11 fault 	 0x2845 : 	 0x2850 : 	 11  
aida12 fault 	 0x1bf6 : 	 0x1c01 : 	 11  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

Also errors in FPGA
			 Base 		Current 		Difference
aida03 fault 	 0x0 : 	 0x1 : 	 1  
aida04 fault 	 0x0 : 	 0x1 : 	 1  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x1 : 	 0x2 : 	 1  
FPGA Timestamp error counter test result: Passed 7, Failed 5
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A8D22 3A5FDE66 , WR/10=>   23DDAE9D2A32FD7, Readout Time =>   23DDAE9D3C9C000 
aida02 : White Rabbit=>  166A8D22 4C6C3AEB , WR/10=>   23DDAE9D4713917, Readout Time =>   23DDAE9D5B94000 
aida03 : White Rabbit=>  166A8D22 5FB33A3C , WR/10=>   23DDAE9D65EB906, Readout Time =>   23DDAE9D769C000 
aida04 : White Rabbit=>  166A8D22 6FBA29DB , WR/10=>   23DDAE9D7F9042F, Readout Time =>   23DDAE9D8D14000 
aida05 : White Rabbit=>  166A8D22 7F71248D , WR/10=>   23DDAE9D98B5074, Readout Time =>   23DDAE9DA9D8000 
aida06 : White Rabbit=>  166A8D22 8F78B071 , WR/10=>   23DDAE9DB25AB3E, Readout Time =>   23DDAE9DC280000 
aida07 : White Rabbit=>  166A8D22 9F5A64B7 , WR/10=>   23DDAE9DCBC3D45, Readout Time =>   23DDAE9DDFF0000 
aida08 : White Rabbit=>  166A8D22 B1E91092 , WR/10=>   23DDAE9DE974E75, Readout Time =>   23DDAE9DFA08000 
aida09 : White Rabbit=>  166A8D22 C150D9ED , WR/10=>   23DDAE9E021AF64, Readout Time =>   23DDAE9E0FD8000 
aida10 : White Rabbit=>  166A8D22 D02BA583 , WR/10=>   23DDAE9E19DF6F3, Readout Time =>   23DDAE9E2D50000 
aida11 : White Rabbit=>  166A8D22 E287CFC2 , WR/10=>   23DDAE9E373FB2D, Readout Time =>   23DDAE9E4720000 
aida12 : White Rabbit=>  166A8D22 F2D4C911 , WR/10=>   23DDAE9E515474E, Readout Time =>   23DDAE9E1BCC000 

System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results

AIDA DAQ Stopped (Still no beam)
The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA).

03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully.

All system wide checks in AIDA now pass. (AIDA06 no longer shows the error)
Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again.

05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7
      The problem was with the accelerator operators system. There was no beam.

06:20 Error in WR system wide checks. Rest all ok
      This time there is no bad merge items recorded in the merger terminal.
      There are also no timewarp events seen in ucesb which were observed with the issues at 3am.

		 Base 		Current 	Difference
aida01 fault 	 0xdbe9 : 	 0xdbea : 	 1  
aida02 fault 	 0xa93d : 	 0xa93e : 	 1  
aida03 fault 	 0x8fc4 : 	 0x8fc5 : 	 1  
aida04 fault 	 0x3e56 : 	 0x3e57 : 	 1  
aida05 fault 	 0xb04c : 	 0xb04d : 	 1  
aida06 fault 	 0x5576 : 	 0x5577 : 	 1  
aida07 fault 	 0x99ff : 	 0x9a00 : 	 1  
aida08 fault 	 0x8412 : 	 0x8413 : 	 1  
aida09 fault 	 0x2457 : 	 0x2458 : 	 1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A9523 11478D1D , WR/10=>   23DDBB6B4ED8E1C, Readout Time =>   23DDBB6B6148000 
aida02 : White Rabbit=>  166A9523 236AA5E8 , WR/10=>   23DDBB6B6BDDD64, Readout Time =>   23DDBB6B8164000 
aida03 : White Rabbit=>  166A9523 375F235D , WR/10=>   23DDBB6B8BCB6BC, Readout Time =>   23DDBB6B9C3C000 
aida04 : White Rabbit=>  166A9523 47B8038C , WR/10=>   23DDBB6BA5F338E, Readout Time =>   23DDBB6BA77C000 
aida05 : White Rabbit=>  166A9523 599FC116 , WR/10=>   23DDBB6BC29934F, Readout Time =>   23DDBB6BD34C000 
aida06 : White Rabbit=>  166A9523 696EB0A7 , WR/10=>   23DDBB6BDBE44DD, Readout Time =>   23DDBB6BECB8000 
aida07 : White Rabbit=>  166A9523 7982236B , WR/10=>   23DDBB6BF59D057, Readout Time =>   23DDBB6C0418000 
aida08 : White Rabbit=>  166A9523 87C85008 , WR/10=>   23DDBB6C0C73B34, Readout Time =>   23DDBB6C1C7C000 
aida09 : White Rabbit=>  166A9523 97020497 , WR/10=>   23DDBB6C24D0075, Readout Time =>   23DDBB6C34B4000 
aida10 : White Rabbit=>  166A9523 A69B171D , WR/10=>   23DDBB6C3DC4F1C, Readout Time =>   23DDBB6C50A0000 
aida11 : White Rabbit=>  166A9523 B86C8D7D , WR/10=>   23DDBB6C5A4748C, Readout Time =>   23DDBB6C6C88000 
aida12 : White Rabbit=>  166A9523 CA1DD1B2 , WR/10=>   23DDBB6C76961C5, Readout Time =>   23DDBB6C5C40000 

Possibly occured at 05:28 as a ucesb reports a timewarp at the time.
AIDA Timewarp (166a922b75368928 before 166a922b799e6c92)
AIDA timewarp is over, skipped 2728 AIDA event(s) 
Cannot check file as was not writing to file.

      Statistics - attachment 8
      FEE Temperatures - attachment 9
      Bias and leakage currents ok - attachment 10

06:40 There are occasional seconds of beam. According to operators it comes and then it goes again

06:45 Beam fairly stable so we are running again.
      AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19

07:00 Beam stopped again
07:02 Beam returned
      There are apparently issues with MUSIC chamber one and its resolution

07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing. 
Attachment 1: 210308_2354_Stats.png
210308_2354_Stats.png
Attachment 2: 210308_2355_Temp.png
210308_2355_Temp.png
Attachment 3: 210308_2357_Bias.png
210308_2357_Bias.png
Attachment 4: 210309_0104_Stats.png
210309_0104_Stats.png
Attachment 5: 210309_0205_Temp.png
210309_0205_Temp.png
Attachment 6: 210309_0205_Bias.png
210309_0205_Bias.png
Attachment 7: 210309_0529_layout1.png
210309_0529_layout1.png
Attachment 8: 210309_0619_Stats.png
210309_0619_Stats.png
Attachment 9: 210309_0620_Temp.png
210309_0620_Temp.png
Attachment 10: 210309_0623_Bias.png
210309_0623_Bias.png
  181   Mon Mar 8 17:02:38 2021 CAMonday 8th March 18:00 - 00:00
18:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

18:24 Rate spectra - attachment 4

18:30 Merger ok - 4.4M data items/s

      TapeServer ok - 43MB/s

      Data forwarding to MBS ok

19:28 beam back after a brief outage - AIDA on file R9_736

      FRS having DAQ issues

19:57 FRS DAQ/Scalars back

20:12 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 5
      FEE Temperatures - attachment 6
      Bias and leakage currents ok - attachment 7

      Merger ok - 4.5M data items/s

      TapeServer ok - 46MB/s

      Data forwarding to MBS ok

20:33 No beam - AIDA on file R9_821

21:08 rates spectra - attachment 8

22:35 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 9
      FEE Temperatures - attachment 10
      Bias and leakage currents ok - attachment 11

      Merger ok - 4.4M data items/s

      TapeServer ok - 43MB/s

      Data forwarding to MBS ok

      temperatures slightly longer to load, and aida04 came up as not responding. Responded as normal after another reload.

23:07 media/ThirdDrive at 65% capacity

      writing to file R9_1021

      file compression currently up to R1_686

23:36 beam stable ~1e9 pps

      rate spectra - attachment 12




      
Attachment 1: Screenshot_from_2021-03-08_17-05-44.png
Screenshot_from_2021-03-08_17-05-44.png
Attachment 2: Screenshot_from_2021-03-08_17-07-04.png
Screenshot_from_2021-03-08_17-07-04.png
Attachment 3: Screenshot_from_2021-03-08_17-08-51.png
Screenshot_from_2021-03-08_17-08-51.png
Attachment 4: Screenshot_from_2021-03-08_17-23-58.png
Screenshot_from_2021-03-08_17-23-58.png
Attachment 5: Screenshot_from_2021-03-08_19-10-56.png
Screenshot_from_2021-03-08_19-10-56.png
Attachment 6: Screenshot_from_2021-03-08_19-11-33.png
Screenshot_from_2021-03-08_19-11-33.png
Attachment 7: Screenshot_from_2021-03-08_19-11-53.png
Screenshot_from_2021-03-08_19-11-53.png
Attachment 8: Screenshot_from_2021-03-08_20-08-12.png
Screenshot_from_2021-03-08_20-08-12.png
Attachment 9: Screenshot_from_2021-03-08_21-28-29.png
Screenshot_from_2021-03-08_21-28-29.png
Attachment 10: Screenshot_from_2021-03-08_21-29-08.png
Screenshot_from_2021-03-08_21-29-08.png
Attachment 11: Screenshot_from_2021-03-08_21-32-35.png
Screenshot_from_2021-03-08_21-32-35.png
Attachment 12: Screenshot_from_2021-03-08_22-38-41.png
Screenshot_from_2021-03-08_22-38-41.png
  180   Mon Mar 8 09:45:20 2021 CA, TDAnalysis R7_20 (new version of NewMerger with min info code 4 & 5 data items)
An analysis of file R6_70 can be found at https://elog.ph.ed.ac.uk/DESPEC/177 attachment 19
We observe c. 80M ADC data items per 2Gb file and c. 160M info code 4 & 5 data items.

This am NH switched NewMerger for a new version which minimises the number of info code 4 & 5
data items. The analysis of one of these data files (R7_20) is appended - attachment 1. We
observe c. 2M ADC data items per 2Gb file. The number of info code 4 data items is significantly
reduced as expected but info code 5 data items are not observed.

Info code 4 & 5 data items will increment every (2^28 * 1e-9) 0.268s and (2^48 * 1e-9) 3.26d.

If we examine the verbose output (attachment 2) we observe that each data block only contains
c. 60 ADC data items - a small fraction of the capacity of each data block.  
Attachment 1: R7_20
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:    1822115 ( 2111176.3 Hz)
        Other data format:       1885 (    2184.0 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:          8 (       9.3 Hz)
                               RESUME:         10 (      11.6 Hz)
                              SYNC100:          4 (       4.6 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:       1857 (    2151.6 Hz)
                             MBS info:          6 (       7.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:          0 (       0.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:        0.863 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.206                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.060                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                0.201                0.000
  9                0.166                0.805
 10                0.000                0.000
 11                0.592                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     150071         10          0          0          2          2          0          0          0          6          0          0
  1     194013          1          0          0          0          0          1          0          0          0          0          0
  2     150373       1859          0          0          1          1          0          0       1857          0          0          0
  3     230996          0          0          0          0          0          0          0          0          0          0          0
  4      53452          0          0          0          0          0          0          0          0          0          0          0
  5     142131          5          0          0          2          3          0          0          0          0          0          0
  6     178856          0          0          0          0          0          0          0          0          0          0          0
  7      74144          0          0          0          0          0          0          0          0          0          0          0
  8     183174          3          0          0          1          2          0          0          0          0          0          0
  9     218137          5          0          0          1          1          3          0          0          0          0          0
 10      83482          0          0          0          0          0          0          0          0          0          0          0
 11     163286          2          0          0          1          1          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   23.008s ( 1390.832 blocks/s,  86.927 Mb/s)
Attachment 2: R7_20_verbose
[npg@aidas-gsi analyser]$ ./analyser v  /TapeData/S452/R7_20 | more
 *** TDR format 3.3.0 analyser - TD - January 2019
  verbose
 ***            ADC data: block:     1 ptr:     7 data: 0xC1177F65 module:  4 fail: 0 range: 0 id:   279 channel: 23 adc: 32613  ts: 0x00000000078DEB0E
 ***            ADC data: block:     1 ptr:     9 data: 0xC0478143 module:  1 fail: 0 range: 0 id:    71 channel:  7 adc: 33091  ts: 0x00000000078DEB4A
 ***            ADC data: block:     1 ptr:    11 data: 0xC0D182FC module:  3 fail: 0 range: 0 id:   209 channel: 17 adc: 33532  ts: 0x00000000078DEB68
 ***            ADC data: block:     1 ptr:    13 data: 0xC2C081B9 module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 33209  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    15 data: 0xC2D7820D module: 11 fail: 0 range: 0 id:   727 channel: 23 adc: 33293  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    17 data: 0xC2E4835E module: 11 fail: 0 range: 0 id:   740 channel: 36 adc: 33630  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    19 data: 0xC2F681F4 module: 11 fail: 0 range: 0 id:   758 channel: 54 adc: 33268  ts: 0x00000000078DF068
 ***            ADC data: block:     1 ptr:    21 data: 0xC00B7FDF module:  0 fail: 0 range: 0 id:    11 channel: 11 adc: 32735  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    23 data: 0xC0147DDE module:  0 fail: 0 range: 0 id:    20 channel: 20 adc: 32222  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    25 data: 0xC034803A module:  0 fail: 0 range: 0 id:    52 channel: 52 adc: 32826  ts: 0x00000000078DF1B2
 ***            ADC data: block:     1 ptr:    27 data: 0xC272814E module:  9 fail: 0 range: 0 id:   626 channel: 50 adc: 33102  ts: 0x00000000078DF2A2
 ***            ADC data: block:     1 ptr:    29 data: 0xC11B7E13 module:  4 fail: 0 range: 0 id:   283 channel: 27 adc: 32275  ts: 0x00000000078DF2DE
 ***            ADC data: block:     1 ptr:    31 data: 0xC1327D8E module:  4 fail: 0 range: 0 id:   306 channel: 50 adc: 32142  ts: 0x00000000078DF2DE
 ***            ADC data: block:     1 ptr:    33 data: 0xC05F7FB9 module:  1 fail: 0 range: 0 id:    95 channel: 31 adc: 32697  ts: 0x00000000078DF31A
 ***            ADC data: block:     1 ptr:    35 data: 0xC0C28087 module:  3 fail: 0 range: 0 id:   194 channel:  2 adc: 32903  ts: 0x00000000078DF338
 ***            ADC data: block:     1 ptr:    37 data: 0xC0DB8075 module:  3 fail: 0 range: 0 id:   219 channel: 27 adc: 32885  ts: 0x00000000078DF338
 ***            ADC data: block:     1 ptr:    39 data: 0xC1BA7ECE module:  6 fail: 0 range: 0 id:   442 channel: 58 adc: 32462  ts: 0x00000000078DF644
 ***            ADC data: block:     1 ptr:    41 data: 0xC2C381FB module: 11 fail: 0 range: 0 id:   707 channel:  3 adc: 33275  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    43 data: 0xC2D981FD module: 11 fail: 0 range: 0 id:   729 channel: 25 adc: 33277  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    45 data: 0xC2E682A5 module: 11 fail: 0 range: 0 id:   742 channel: 38 adc: 33445  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    47 data: 0xC2F782E0 module: 11 fail: 0 range: 0 id:   759 channel: 55 adc: 33504  ts: 0x00000000078DF838
 ***            ADC data: block:     1 ptr:    49 data: 0xC0177CE4 module:  0 fail: 0 range: 0 id:    23 channel: 23 adc: 31972  ts: 0x00000000078DF982
 ***            ADC data: block:     1 ptr:    51 data: 0xC27A7F8E module:  9 fail: 0 range: 0 id:   634 channel: 58 adc: 32654  ts: 0x00000000078DFA72
 ***            ADC data: block:     1 ptr:    53 data: 0xC0C3816A module:  3 fail: 0 range: 0 id:   195 channel:  3 adc: 33130  ts: 0x00000000078DFB08
 ***            ADC data: block:     1 ptr:    55 data: 0xC0FD81FD module:  3 fail: 0 range: 0 id:   253 channel: 61 adc: 33277  ts: 0x00000000078DFB08
 ***            ADC data: block:     1 ptr:    57 data: 0xC0847F9A module:  2 fail: 0 range: 0 id:   132 channel:  4 adc: 32666  ts: 0x00000000078DFCB6
 ***            ADC data: block:     1 ptr:    59 data: 0xC0B77EED module:  2 fail: 0 range: 0 id:   183 channel: 55 adc: 32493  ts: 0x00000000078DFCB6
 ***            ADC data: block:     1 ptr:    61 data: 0xC1AD8002 module:  6 fail: 0 range: 0 id:   429 channel: 45 adc: 32770  ts: 0x00000000078DFE14
 ***            ADC data: block:     1 ptr:    63 data: 0xC2C4837A module: 11 fail: 0 range: 0 id:   708 channel:  4 adc: 33658  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    65 data: 0xC2DC8250 module: 11 fail: 0 range: 0 id:   732 channel: 28 adc: 33360  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    67 data: 0xC2EA8214 module: 11 fail: 0 range: 0 id:   746 channel: 42 adc: 33300  ts: 0x00000000078E0008
 ***            ADC data: block:     1 ptr:    69 data: 0xC27B812C module:  9 fail: 0 range: 0 id:   635 channel: 59 adc: 33068  ts: 0x00000000078E0242
 ***            ADC data: block:     1 ptr:    71 data: 0xC0C5811F module:  3 fail: 0 range: 0 id:   197 channel:  5 adc: 33055  ts: 0x00000000078E02D8
 ***            ADC data: block:     1 ptr:    73 data: 0xC0877F4D module:  2 fail: 0 range: 0 id:   135 channel:  7 adc: 32589  ts: 0x00000000078E0486
 ***            ADC data: block:     1 ptr:    75 data: 0xC2C782D6 module: 11 fail: 0 range: 0 id:   711 channel:  7 adc: 33494  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    77 data: 0xC2DE8261 module: 11 fail: 0 range: 0 id:   734 channel: 30 adc: 33377  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    79 data: 0xC2EB8115 module: 11 fail: 0 range: 0 id:   747 channel: 43 adc: 33045  ts: 0x00000000078E07D8
 ***            ADC data: block:     1 ptr:    81 data: 0xC0027C33 module:  0 fail: 0 range: 0 id:     2 channel:  2 adc: 31795  ts: 0x00000000078E0922
 ***            ADC data: block:     1 ptr:    83 data: 0xC0C68218 module:  3 fail: 0 range: 0 id:   198 channel:  6 adc: 33304  ts: 0x00000000078E0AA8
 ***            ADC data: block:     1 ptr:    85 data: 0xC0D58287 module:  3 fail: 0 range: 0 id:   213 channel: 21 adc: 33415  ts: 0x00000000078E0AA8
 ***            ADC data: block:     1 ptr:    87 data: 0xC08C4EFE module:  2 fail: 0 range: 0 id:   140 channel: 12 adc: 20222  ts: 0x00000000078E0C56
 ***            ADC data: block:     1 ptr:    89 data: 0xC2F48203 module: 11 fail: 0 range: 0 id:   756 channel: 52 adc: 33283  ts: 0x00000000078E0FA8
 ***            ADC data: block:     1 ptr:    91 data: 0xC267809D module:  9 fail: 0 range: 0 id:   615 channel: 39 adc: 32925  ts: 0x00000000078E11E2
 ***            ADC data: block:     1 ptr:    93 data: 0xC044810C module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 33036  ts: 0x00000000078E125A
 ***            ADC data: block:     1 ptr:    95 data: 0xC0C9815B module:  3 fail: 0 range: 0 id:   201 channel:  9 adc: 33115  ts: 0x00000000078E1278
 ***            ADC data: block:     1 ptr:    97 data: 0xC0DA8174 module:  3 fail: 0 range: 0 id:   218 channel: 26 adc: 33140  ts: 0x00000000078E1278
 ***            ADC data: block:     1 ptr:    99 data: 0xC1827F8D module:  6 fail: 0 range: 0 id:   386 channel:  2 adc: 32653  ts: 0x00000000078E1584
 ***            ADC data: block:     1 ptr:   101 data: 0xC1A77E56 module:  6 fail: 0 range: 0 id:   423 channel: 39 adc: 32342  ts: 0x00000000078E1584
 ***            ADC data: block:     1 ptr:   103 data: 0xC2FD846E module: 11 fail: 0 range: 0 id:   765 channel: 61 adc: 33902  ts: 0x00000000078E1778
 ***            ADC data: block:     1 ptr:   105 data: 0xC27081A7 module:  9 fail: 0 range: 0 id:   624 channel: 48 adc: 33191  ts: 0x00000000078E19B2
 ***            ADC data: block:     1 ptr:   107 data: 0xC0CA8171 module:  3 fail: 0 range: 0 id:   202 channel: 10 adc: 33137  ts: 0x00000000078E1A48
 ***            ADC data: block:     1 ptr:   109 data: 0xC0DF80F0 module:  3 fail: 0 range: 0 id:   223 channel: 31 adc: 33008  ts: 0x00000000078E1A48
 ***            ADC data: block:     1 ptr:   111 data: 0xC1F28069 module:  7 fail: 0 range: 0 id:   498 channel: 50 adc: 32873  ts: 0x00000000078E1A7A
 ***            ADC data: block:     1 ptr:   113 data: 0xC1837E89 module:  6 fail: 0 range: 0 id:   387 channel:  3 adc: 32393  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   115 data: 0xC1987EC6 module:  6 fail: 0 range: 0 id:   408 channel: 24 adc: 32454  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   117 data: 0xC1B37F93 module:  6 fail: 0 range: 0 id:   435 channel: 51 adc: 32659  ts: 0x00000000078E1D54
 ***            ADC data: block:     1 ptr:   119 data: 0xC2C28169 module: 11 fail: 0 range: 0 id:   706 channel:  2 adc: 33129  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:     7 data: 0xC2D5830E module: 11 fail: 0 range: 0 id:   725 channel: 21 adc: 33550  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:     9 data: 0xC2E18150 module: 11 fail: 0 range: 0 id:   737 channel: 33 adc: 33104  ts: 0x00000000078E1F48
 ***            ADC data: block:     2 ptr:    11 data: 0xC0117F5D module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 32605  ts: 0x00000000078E2092
 ***            ADC data: block:     2 ptr:    13 data: 0xC27E81BD module:  9 fail: 0 range: 0 id:   638 channel: 62 adc: 33213  ts: 0x00000000078E2182
 ***            ADC data: block:     2 ptr:    15 data: 0xC07D817F module:  1 fail: 0 range: 0 id:   125 channel: 61 adc: 33151  ts: 0x00000000078E21FA
 ***            ADC data: block:     2 ptr:    17 data: 0xC0CB8125 module:  3 fail: 0 range: 0 id:   203 channel: 11 adc: 33061  ts: 0x00000000078E2218
 ***            ADC data: block:     2 ptr:    19 data: 0xC1C28219 module:  7 fail: 0 range: 0 id:   450 channel:  2 adc: 33305  ts: 0x00000000078E224A
 ***            ADC data: block:     2 ptr:    21 data: 0xC1F77FD8 module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 32728  ts: 0x00000000078E224A
 ***            ADC data: block:     2 ptr:    23 data: 0xC08B7FBA module:  2 fail: 0 range: 0 id:   139 channel: 11 adc: 32698  ts: 0x00000000078E23C6
 ***            ADC data: block:     2 ptr:    25 data: 0xC09F7E74 module:  2 fail: 0 range: 0 id:   159 channel: 31 adc: 32372  ts: 0x00000000078E23C6
 ***            ADC data: block:     2 ptr:    27 data: 0xC1867FE8 module:  6 fail: 0 range: 0 id:   390 channel:  6 adc: 32744  ts: 0x00000000078E2524
 ***            ADC data: block:     2 ptr:    29 data: 0xC19A7EC3 module:  6 fail: 0 range: 0 id:   410 channel: 26 adc: 32451  ts: 0x00000000078E2524
 ***            ADC data: block:     2 ptr:    31 data: 0xC2C98098 module: 11 fail: 0 range: 0 id:   713 channel:  9 adc: 32920  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    33 data: 0xC2DA837D module: 11 fail: 0 range: 0 id:   730 channel: 26 adc: 33661  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    35 data: 0xC2E582C0 module: 11 fail: 0 range: 0 id:   741 channel: 37 adc: 33472  ts: 0x00000000078E2718
 ***            ADC data: block:     2 ptr:    37 data: 0xC0CC81D9 module:  3 fail: 0 range: 0 id:   204 channel: 12 adc: 33241  ts: 0x00000000078E29E8
 ***            ADC data: block:     2 ptr:    39 data: 0xC187804E module:  6 fail: 0 range: 0 id:   391 channel:  7 adc: 32846  ts: 0x00000000078E2CF4
 ***            ADC data: block:     2 ptr:    41 data: 0xC2DB81B9 module: 11 fail: 0 range: 0 id:   731 channel: 27 adc: 33209  ts: 0x00000000078E2EE8
 ***            ADC data: block:     2 ptr:    43 data: 0xC2E782E0 module: 11 fail: 0 range: 0 id:   743 channel: 39 adc: 33504  ts: 0x00000000078E2EE8
 ***            ADC data: block:     2 ptr:    45 data: 0xC0CD80DF module:  3 fail: 0 range: 0 id:   205 channel: 13 adc: 32991  ts: 0x00000000078E31B8
 ***            ADC data: block:     2 ptr:    47 data: 0xC0D380C6 module:  3 fail: 0 range: 0 id:   211 channel: 19 adc: 32966  ts: 0x00000000078E31B8
 ***            ADC data: block:     2 ptr:    49 data: 0xC1897EEF module:  6 fail: 0 range: 0 id:   393 channel:  9 adc: 32495  ts: 0x00000000078E34C4
 ***            ADC data: block:     2 ptr:    51 data: 0xC2E98401 module: 11 fail: 0 range: 0 id:   745 channel: 41 adc: 33793  ts: 0x00000000078E36B8
 ***            ADC data: block:     2 ptr:    53 data: 0xC0E08147 module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 33095  ts: 0x00000000078E3988
 ***            ADC data: block:     2 ptr:    55 data: 0xC2C881AC module: 11 fail: 0 range: 0 id:   712 channel:  8 adc: 33196  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    57 data: 0xC2ED8349 module: 11 fail: 0 range: 0 id:   749 channel: 45 adc: 33609  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    59 data: 0xC2F380C2 module: 11 fail: 0 range: 0 id:   755 channel: 51 adc: 32962  ts: 0x00000000078E3E88
 ***            ADC data: block:     2 ptr:    61 data: 0xC1BD7C77 module:  6 fail: 0 range: 0 id:   445 channel: 61 adc: 31863  ts: 0x00000000078E4464
 ***            ADC data: block:     2 ptr:    63 data: 0xC2D3837F module: 11 fail: 0 range: 0 id:   723 channel: 19 adc: 33663  ts: 0x00000000078E4658
 ***            ADC data: block:     2 ptr:    65 data: 0xC24F80FC module:  9 fail: 0 range: 0 id:   591 channel: 15 adc: 33020  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    67 data: 0xC26C80FA module:  9 fail: 0 range: 0 id:   620 channel: 44 adc: 33018  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    69 data: 0xC27782CE module:  9 fail: 0 range: 0 id:   631 channel: 55 adc: 33486  ts: 0x00000000078E4892
 ***            ADC data: block:     2 ptr:    71 data: 0xC0C08052 module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 32850  ts: 0x00000000078E4928
 ***            ADC data: block:     2 ptr:    73 data: 0xC08E4C1D module:  2 fail: 0 range: 0 id:   142 channel: 14 adc: 19485  ts: 0x00000000078E4AD6
 ***            ADC data: block:     2 ptr:    75 data: 0xC1817D36 module:  6 fail: 0 range: 0 id:   385 channel:  1 adc: 32054  ts: 0x00000000078E4C34
 ***            ADC data: block:     2 ptr:    77 data: 0xC1BF7FAF module:  6 fail: 0 range: 0 id:   447 channel: 63 adc: 32687  ts: 0x00000000078E4C34
 ***            ADC data: block:     2 ptr:    79 data: 0xC2E2826E module: 11 fail: 0 range: 0 id:   738 channel: 34 adc: 33390  ts: 0x00000000078E55F8
 ***            ADC data: block:     2 ptr:    81 data: 0xC1C380E1 module:  7 fail: 0 range: 0 id:   451 channel:  3 adc: 32993  ts: 0x00000000078E58FA
 ***            ADC data: block:     2 ptr:    83 data: 0xC0F78257 module:  3 fail: 0 range: 0 id:   247 channel: 55 adc: 33367  ts: 0x00000000078E6098
 ***            ADC data: block:     2 ptr:    85 data: 0xC2C681D1 module: 11 fail: 0 range: 0 id:   710 channel:  6 adc: 33233  ts: 0x00000000078E6598
 ***            ADC data: block:     2 ptr:    87 data: 0xC04E8051 module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 32849  ts: 0x00000000078E684A
 ***            ADC data: block:     2 ptr:    89 data: 0xC0758189 module:  1 fail: 0 range: 0 id:   117 channel: 53 adc: 33161  ts: 0x00000000078E684A
 ***            ADC data: block:     2 ptr:    91 data: 0xC0948066 module:  2 fail: 0 range: 0 id:   148 channel: 20 adc: 32870  ts: 0x00000000078E6A16
 ***            ADC data: block:     2 ptr:    93 data: 0xC0057DFD module:  0 fail: 0 range: 0 id:     5 channel:  5 adc: 32253  ts: 0x00000000078E6EB2
 ***            ADC data: block:     2 ptr:    95 data: 0xC2418102 module:  9 fail: 0 range: 0 id:   577 channel:  1 adc: 33026  ts: 0x00000000078E6FA2
 ***            ADC data: block:     2 ptr:    97 data: 0xC26B8030 module:  9 fail: 0 range: 0 id:   619 channel: 43 adc: 32816  ts: 0x00000000078E6FA2
 ***            ADC data: block:     2 ptr:    99 data: 0xC0077C68 module:  0 fail: 0 range: 0 id:     7 channel:  7 adc: 31848  ts: 0x00000000078E7682
 ***            ADC data: block:     2 ptr:   101 data: 0xC0107DBC module:  0 fail: 0 range: 0 id:    16 channel: 16 adc: 32188  ts: 0x00000000078E7682
 ***            ADC data: block:     2 ptr:   103 data: 0xC26F8145 module:  9 fail: 0 range: 0 id:   623 channel: 47 adc: 33093  ts: 0x00000000078E7772
 ***            ADC data: block:     2 ptr:   105 data: 0xC0D18120 module:  3 fail: 0 range: 0 id:   209 channel: 17 adc: 33056  ts: 0x00000000078E7808
 ***            ADC data: block:     2 ptr:   107 data: 0xC2C081F4 module: 11 fail: 0 range: 0 id:   704 channel:  0 adc: 33268  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   109 data: 0xC2D78240 module: 11 fail: 0 range: 0 id:   727 channel: 23 adc: 33344  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   111 data: 0xC2E48241 module: 11 fail: 0 range: 0 id:   740 channel: 36 adc: 33345  ts: 0x00000000078E7D08
 ***            ADC data: block:     2 ptr:   113 data: 0xC0137D35 module:  0 fail: 0 range: 0 id:    19 channel: 19 adc: 32053  ts: 0x00000000078E7E52
 ***            ADC data: block:     2 ptr:   115 data: 0xC2728164 module:  9 fail: 0 range: 0 id:   626 channel: 50 adc: 33124  ts: 0x00000000078E7F42
 ***            ADC data: block:     2 ptr:   117 data: 0xC0C18052 module:  3 fail: 0 range: 0 id:   193 channel:  1 adc: 32850  ts: 0x00000000078E7FD8
 ***            ADC data: block:     2 ptr:   119 data: 0xC2C38244 module: 11 fail: 0 range: 0 id:   707 channel:  3 adc: 33348  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:     7 data: 0xC2D982DC module: 11 fail: 0 range: 0 id:   729 channel: 25 adc: 33500  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:     9 data: 0xC2E683C3 module: 11 fail: 0 range: 0 id:   742 channel: 38 adc: 33731  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:    11 data: 0xC2F68233 module: 11 fail: 0 range: 0 id:   758 channel: 54 adc: 33331  ts: 0x00000000078E84D8
 ***            ADC data: block:     3 ptr:    13 data: 0xC0177D5C module:  0 fail: 0 range: 0 id:    23 channel: 23 adc: 32092  ts: 0x00000000078E8622
 ***            ADC data: block:     3 ptr:    15 data: 0xC0C2803E module:  3 fail: 0 range: 0 id:   194 channel:  2 adc: 32830  ts: 0x00000000078E87A8
 ***            ADC data: block:     3 ptr:    17 data: 0xC0FD81FC module:  3 fail: 0 range: 0 id:   253 channel: 61 adc: 33276  ts: 0x00000000078E87A8
 ***            ADC data: block:     3 ptr:    19 data: 0xC1AD7F36 module:  6 fail: 0 range: 0 id:   429 channel: 45 adc: 32566  ts: 0x00000000078E8AB4
 ***            ADC data: block:     3 ptr:    21 data: 0xC2F78642 module: 11 fail: 0 range: 0 id:   759 channel: 55 adc: 34370  ts: 0x00000000078E8CA8
 ***            ADC data: block:     3 ptr:    23 data: 0xC0C3815B module:  3 fail: 0 range: 0 id:   195 channel:  3 adc: 33115  ts: 0x00000000078E8F78
 ***            ADC data: block:     3 ptr:    25 data: 0xC0847EE9 module:  2 fail: 0 range: 0 id:   132 channel:  4 adc: 32489  ts: 0x00000000078E9126
 ***            ADC data: block:     3 ptr:    27 data: 0xC0B77F35 module:  2 fail: 0 range: 0 id:   183 channel: 55 adc: 32565  ts: 0x00000000078E9126
 ***            ADC data: block:     3 ptr:    29 data: 0xC27B8125 module:  9 fail: 0 range: 0 id:   635 channel: 59 adc: 33061  ts: 0x00000000078E96B2
 ***            ADC data: block:     3 ptr:    31 data: 0xC08C3881 module:  2 fail: 0 range: 0 id:   140 channel: 12 adc: 14465  ts: 0x00000000078E98F6
 ***            ADC data: block:     3 ptr:    33 data: 0xC2C181D1 module: 11 fail: 0 range: 0 id:   705 channel:  1 adc: 33233  ts: 0x00000000078E9C48
 ***            ADC data: block:     3 ptr:    35 data: 0xC2D480BF module: 11 fail: 0 range: 0 id:   724 channel: 20 adc: 32959  ts: 0x00000000078E9C48
 ***            ADC data: block:     3 ptr:    37 data: 0xC2648191 module:  9 fail: 0 range: 0 id:   612 channel: 36 adc: 33169  ts: 0x00000000078E9E82
 ***            ADC data: block:     3 ptr:    39 data: 0xC04780C3 module:  1 fail: 0 range: 0 id:    71 channel:  7 adc: 32963  ts: 0x00000000078E9EFA
 ***            ADC data: block:     3 ptr:    41 data: 0xC0E6809B module:  3 fail: 0 range: 0 id:   230 channel: 38 adc: 32923  ts: 0x00000000078E9F18
 ***            ADC data: block:     3 ptr:    43 data: 0xC2C4834F module: 11 fail: 0 range: 0 id:   708 channel:  4 adc: 33615  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    45 data: 0xC2DC8132 module: 11 fail: 0 range: 0 id:   732 channel: 28 adc: 33074  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    47 data: 0xC2FD8325 module: 11 fail: 0 range: 0 id:   765 channel: 61 adc: 33573  ts: 0x00000000078EA418
 ***            ADC data: block:     3 ptr:    49 data: 0xC0C581D9 module:  3 fail: 0 range: 0 id:   197 channel:  5 adc: 33241  ts: 0x00000000078EA6E8
 ***            ADC data: block:     3 ptr:    51 data: 0xC2E183D2 module: 11 fail: 0 range: 0 id:   737 channel: 33 adc: 33746  ts: 0x00000000078EABE8
 ***            ADC data: block:     3 ptr:    53 data: 0xC0C9806C module:  3 fail: 0 range: 0 id:   201 channel:  9 adc: 32876  ts: 0x00000000078EAEB8
 ***            ADC data: block:     3 ptr:    55 data: 0xC0DA814C module:  3 fail: 0 range: 0 id:   218 channel: 26 adc: 33100  ts: 0x00000000078EAEB8
 ***            ADC data: block:     3 ptr:    57 data: 0xC1C28260 module:  7 fail: 0 range: 0 id:   450 channel:  2 adc: 33376  ts: 0x00000000078EAEEA
 ***            ADC data: block:     3 ptr:    59 data: 0xC08B7FEC module:  2 fail: 0 range: 0 id:   139 channel: 11 adc: 32748  ts: 0x00000000078EB066
 ***            ADC data: block:     3 ptr:    61 data: 0xC09F7EF1 module:  2 fail: 0 range: 0 id:   159 channel: 31 adc: 32497  ts: 0x00000000078EB066
 ***            ADC data: block:     3 ptr:    63 data: 0xC2668171 module:  9 fail: 0 range: 0 id:   614 channel: 38 adc: 33137  ts: 0x00000000078EB5F2
 ***            ADC data: block:     3 ptr:    65 data: 0xC0CA8094 module:  3 fail: 0 range: 0 id:   202 channel: 10 adc: 32916  ts: 0x00000000078EB688
 ***            ADC data: block:     3 ptr:    67 data: 0xC0DF806C module:  3 fail: 0 range: 0 id:   223 channel: 31 adc: 32876  ts: 0x00000000078EB688
 ***            ADC data: block:     3 ptr:    69 data: 0xC1F782AB module:  7 fail: 0 range: 0 id:   503 channel: 55 adc: 33451  ts: 0x00000000078EB6BA
 ***            ADC data: block:     3 ptr:    71 data: 0xC2C282CB module: 11 fail: 0 range: 0 id:   706 channel:  2 adc: 33483  ts: 0x00000000078EBB88
 ***            ADC data: block:     3 ptr:    73 data: 0xC07D8111 module:  1 fail: 0 range: 0 id:   125 channel: 61 adc: 33041  ts: 0x00000000078EBE3A
 ***            ADC data: block:     3 ptr:    75 data: 0xC0CB81B8 module:  3 fail: 0 range: 0 id:   203 channel: 11 adc: 33208  ts: 0x00000000078EBE58
 ***            ADC data: block:     3 ptr:    77 data: 0xC2E98262 module: 11 fail: 0 range: 0 id:   745 channel: 41 adc: 33378  ts: 0x00000000078EC358
 ***            ADC data: block:     3 ptr:    79 data: 0xC044807F module:  1 fail: 0 range: 0 id:    68 channel:  4 adc: 32895  ts: 0x00000000078EC60A
 ***            ADC data: block:     3 ptr:    81 data: 0xC19A7DA3 module:  6 fail: 0 range: 0 id:   410 channel: 26 adc: 32163  ts: 0x00000000078EC934
 ***            ADC data: block:     3 ptr:    83 data: 0xC2ED8207 module: 11 fail: 0 range: 0 id:   749 channel: 45 adc: 33287  ts: 0x00000000078ECB28
 ***            ADC data: block:     3 ptr:    85 data: 0xC2F380FC module: 11 fail: 0 range: 0 id:   755 channel: 51 adc: 33020  ts: 0x00000000078ECB28
 ***            ADC data: block:     3 ptr:    87 data: 0xC2708152 module:  9 fail: 0 range: 0 id:   624 channel: 48 adc: 33106  ts: 0x00000000078ECD62
 ***            ADC data: block:     3 ptr:    89 data: 0xC1B47F47 module:  6 fail: 0 range: 0 id:   436 channel: 52 adc: 32583  ts: 0x00000000078ED104
 ***            ADC data: block:     3 ptr:    91 data: 0xC2C88150 module: 11 fail: 0 range: 0 id:   712 channel:  8 adc: 33104  ts: 0x00000000078ED2F8
 ***            ADC data: block:     3 ptr:    93 data: 0xC27781B4 module:  9 fail: 0 range: 0 id:   631 channel: 55 adc: 33204  ts: 0x00000000078ED532
 ***            ADC data: block:     3 ptr:    95 data: 0xC0C081FF module:  3 fail: 0 range: 0 id:   192 channel:  0 adc: 33279  ts: 0x00000000078ED5C8
 ***            ADC data: block:     3 ptr:    97 data: 0xC1C780DD module:  7 fail: 0 range: 0 id:   455 channel:  7 adc: 32989  ts: 0x00000000078ED5FA
 ***            ADC data: block:     3 ptr:    99 data: 0xC1BD7D53 module:  6 fail: 0 range: 0 id:   445 channel: 61 adc: 32083  ts: 0x00000000078ED8D4
 ***            ADC data: block:     3 ptr:   101 data: 0xC2D38300 module: 11 fail: 0 range: 0 id:   723 channel: 19 adc: 33536  ts: 0x00000000078EDAC8
 ***            ADC data: block:     3 ptr:   103 data: 0xC0027F2E module:  0 fail: 0 range: 0 id:     2 channel:  2 adc: 32558  ts: 0x00000000078EDC12
 ***            ADC data: block:     3 ptr:   105 data: 0xC0117FD8 module:  0 fail: 0 range: 0 id:    17 channel: 17 adc: 32728  ts: 0x00000000078EDC12
 ***            ADC data: block:     3 ptr:   107 data: 0xC2588009 module:  9 fail: 0 range: 0 id:   600 channel: 24 adc: 32777  ts: 0x00000000078EDD02
 ***            ADC data: block:     3 ptr:   109 data: 0xC1CD810E module:  7 fail: 0 range: 0 id:   461 channel: 13 adc: 33038  ts: 0x00000000078EDDCA
 ***            ADC data: block:     3 ptr:   111 data: 0xC1927F0C module:  6 fail: 0 range: 0 id:   402 channel: 18 adc: 32524  ts: 0x00000000078EE0A4
 ***            ADC data: block:     3 ptr:   113 data: 0xC2DB8197 module: 11 fail: 0 range: 0 id:   731 channel: 27 adc: 33175  ts: 0x00000000078EE298
 ***            ADC data: block:     3 ptr:   115 data: 0xC2E7821D module: 11 fail: 0 range: 0 id:   743 channel: 39 adc: 33309  ts: 0x00000000078EE298
 ***            ADC data: block:     3 ptr:   117 data: 0xC0097F87 module:  0 fail: 0 range: 0 id:     9 channel:  9 adc: 32647  ts: 0x00000000078EE3E2
 ***            ADC data: block:     3 ptr:   119 data: 0xC2598015 module:  9 fail: 0 range: 0 id:   601 channel: 25 adc: 32789  ts: 0x00000000078EE4D2
 ***            ADC data: block:     4 ptr:     7 data: 0xC0E080A6 module:  3 fail: 0 range: 0 id:   224 channel: 32 adc: 32934  ts: 0x00000000078EE568
 ***            ADC data: block:     4 ptr:     9 data: 0xC08E4C7E module:  2 fail: 0 range: 0 id:   142 channel: 14 adc: 19582  ts: 0x00000000078EE716
 ***            ADC data: block:     4 ptr:    11 data: 0xC2E88140 module: 11 fail: 0 range: 0 id:   744 channel: 40 adc: 33088  ts: 0x00000000078EEA68
 ***            ADC data: block:     4 ptr:    13 data: 0xC279812E module:  9 fail: 0 range: 0 id:   633 channel: 57 adc: 33070  ts: 0x00000000078EECA2
 ***            ADC data: block:     4 ptr:    15 data: 0xC0438270 module:  1 fail: 0 range: 0 id:    67 channel:  3 adc: 33392  ts: 0x00000000078EED1A
 ***            ADC data: block:     4 ptr:    17 data: 0xC0D280BB module:  3 fail: 0 range: 0 id:   210 channel: 18 adc: 32955  ts: 0x00000000078EED38
 ***            ADC data: block:     4 ptr:    19 data: 0xC0F78176 module:  3 fail: 0 range: 0 id:   247 channel: 55 adc: 33142  ts: 0x00000000078EED38
 ***            ADC data: block:     4 ptr:    21 data: 0xC1817BAB module:  6 fail: 0 range: 0 id:   385 channel:  1 adc: 31659  ts: 0x00000000078EF044
 ***            ADC data: block:     4 ptr:    23 data: 0xC2F481FD module: 11 fail: 0 range: 0 id:   756 channel: 52 adc: 33277  ts: 0x00000000078EF238
 ***            ADC data: block:     4 ptr:    25 data: 0xC27A7E9E module:  9 fail: 0 range: 0 id:   634 channel: 58 adc: 32414  ts: 0x00000000078EF472
 ***            ADC data: block:     4 ptr:    27 data: 0xC04E80E8 module:  1 fail: 0 range: 0 id:    78 channel: 14 adc: 33000  ts: 0x00000000078EF4EA
 ***            ADC data: block:     4 ptr:    29 data: 0xC0D3809C module:  3 fail: 0 range: 0 id:   211 channel: 19 adc: 32924  ts: 0x00000000078EF508
 ***            ADC data: block:     4 ptr:    31 data: 0xC2C6810C module: 11 fail: 0 range: 0 id:   710 channel:  6 adc: 33036  ts: 0x00000000078EFA08
 ***            ADC data: block:     4 ptr:    33 data: 0xC257808A module:  9 fail: 0 range: 0 id:   599 channel: 23 adc: 32906  ts: 0x00000000078EFC42
 ***            ADC data: block:     4 ptr:    35 data: 0xC0107E1C module:  0 fail: 0 range: 0 id:    16 channel: 16 adc: 32284  ts: 0x00000000078F0322
 ***            ADC data: block:     4 ptr:    37 data: 0xC25A7FA5 module:  9 fail: 0 range: 0 id:   602 channel: 26 adc: 32677  ts: 0x00000000078F0412

[npg@aidas-gsi analyser]$ 

  179   Mon Mar 8 07:02:17 2021 CA, LSMonday 8th March 08:00 - 17:00
08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude 1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

08:20 Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

      Writing to file R6_1148

09:00 Beam off for optimisation - writing to file R6_1212

      Expected back at 09:30

09:30 Beam will be off for a bit longer - an hour or two

09:34 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 4
      FEE Temperatures - attachment 5
      Bias and leakage currents ok - attachment 6

      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

10:10 (NH) Stopped DAQ to try the new merger from VP, new run R7, R8 for these tests
      Tape rate was 110 MB/s (!) and MBS relay seemed nothing... also it didn't stop when DAQ was stopped but took 10+ seconds 
to finish
      Something confusing, will look at data and report to VP
      Reverted to original merger and back... 45 MB/s
      R9

12.00 (LS)
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment7)
      FEE temps (attatchment8)
      Rate spectra (attachment9), beam still down
      Leakage Currents written to sheets (attachment10)
      Merger ~45M items/s
      Tapeserver ~43MB/s

14.00 Still no beam but is expected to return soon
      System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment11)
      FEE temps (attatchment12)
      Leakage Currents written to sheets (attachment13)
      Merger ~45M items/s
      Tapeserver ~44MB/s

14.40 Leakage currents still rising slowly compared to previous days, so will keep an eye on them (attachment14)

15.00 Beam is back (attachment15)
      Still writing to R9 (roughly around R9_386)

15.28 No beam (attachment16)

15.41 Beam back (attachment17)
      Roughly R9 (roughly around R9_439)

16.00 Analysis of R9_469, from FEE0 and FEE2 the hit rate for one side is around 48 HEC data items/second (attachment18)
 
16.20 System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
      Statistics (attachment19)
      Spectra Rate (attachment20)
      FEE temps (attatchment21)
      Leakage Currents written to sheets (attachment22), 
      still rising but looking on Grafana 7 day period should start to lower around 17.00
      Merger ~45M items/s
      Tapeserver ~43MB/s

16.55 Beam down (attachment23), back a couple minutes later

15.51 beam down (attachment24)
      
Attachment 1: Screenshot_from_2021-03-08_07-04-21.png
Screenshot_from_2021-03-08_07-04-21.png
Attachment 2: Screenshot_from_2021-03-08_07-07-24.png
Screenshot_from_2021-03-08_07-07-24.png
Attachment 3: Screenshot_from_2021-03-08_07-07-49.png
Screenshot_from_2021-03-08_07-07-49.png
Attachment 4: Screenshot_from_2021-03-08_08-31-37.png
Screenshot_from_2021-03-08_08-31-37.png
Attachment 5: Screenshot_from_2021-03-08_08-32-58.png
Screenshot_from_2021-03-08_08-32-58.png
Attachment 6: Screenshot_from_2021-03-08_08-34-25.png
Screenshot_from_2021-03-08_08-34-25.png
Attachment 7: 1200_Statistics.PNG
1200_Statistics.PNG
Attachment 8: 1200_FEETemps.PNG
1200_FEETemps.PNG
Attachment 9: 1200_SpectraRate.PNG
1200_SpectraRate.PNG
Attachment 10: 1200_LeakageCurrents.PNG
1200_LeakageCurrents.PNG
Attachment 11: 1400_Statistics.PNG
1400_Statistics.PNG
Attachment 12: 1400_FEETemps.PNG
1400_FEETemps.PNG
Attachment 13: 1400_LeakageCurrents.PNG
1400_LeakageCurrents.PNG
Attachment 14: 1500_Grafana_LeakageCurrents.PNG
1500_Grafana_LeakageCurrents.PNG
Attachment 15: 1500_SpectraRate.PNG
1500_SpectraRate.PNG
Attachment 16: 1528_SpectraRate.PNG
1528_SpectraRate.PNG
Attachment 17: 1541_SpectraRate.PNG
1541_SpectraRate.PNG
Attachment 18: 1600_R9_469_hit_rates.PNG
1600_R9_469_hit_rates.PNG
Attachment 19: 1620_Statistics.PNG
1620_Statistics.PNG
Attachment 20: 1620_SpectraRate.PNG
1620_SpectraRate.PNG
Attachment 21: 1620_FEETemps.PNG
1620_FEETemps.PNG
Attachment 22: 1620_LeakageCurrents.PNG
1620_LeakageCurrents.PNG
Attachment 23: 1655_SpectraRate.PNG
1655_SpectraRate.PNG
Attachment 24: 1749_SpectraRate.PNG
1749_SpectraRate.PNG
  178   Sun Mar 7 23:13:50 2021 OHMonday 8th March
00:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


00:14 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 1
      FEE Temperatures - attachment 2
      Bias and leakage currents ok - attachment 3

01:20 Can see now that we are writing to a separate drive the compression is going much faster - attachment 4

02:15 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 5
      FEE Temperatures - attachment 6
      Bias and leakage currents ok - attachment 7

03:43 Beam has been lost. Not and FRS fault. Several powersupplies have tripped

04:00 Noticed more bad merge events
      System wide checks has also noted an FPGA error.
      All other system wide checks same results as previous

			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks

aida01 : White Rabbit=>  166A3F29 BA0DF4D2 , WR/10=>   23DD31DC5CE3215, Readout Time =>   23DD31DC6F34000 
aida02 : White Rabbit=>  166A3F29 CC49ECD7 , WR/10=>   23DD31DC7A0FE15, Readout Time =>   23DD31DC8428000 
aida03 : White Rabbit=>  166A3F29 DF55B7E8 , WR/10=>   23DD31DC9889264, Readout Time =>   23DD31DCA85C000 
aida04 : White Rabbit=>  166A3F29 EECF82B7 , WR/10=>   23DD31DCB14C045, Readout Time =>   23DD31DCC2B8000 
aida05 : White Rabbit=>  166A3F29 FED2EFD2 , WR/10=>   23DD31DCCAEB195, Readout Time =>   23DD31DCDC14000 
aida06 : White Rabbit=>  166A3F2A 0EDC023D , WR/10=>   23DD31DCE49336C, Readout Time =>   23DD31DCF204000 
aida07 : White Rabbit=>  166A3F2A 1E8170D3 , WR/10=>   23DD31DCFD9BE7B, Readout Time =>   23DD31DD10A0000 
aida08 : White Rabbit=>  166A3F2A 30448F76 , WR/10=>   23DD31DD1A074BF, Readout Time =>   23DD31DD2A34000 
aida09 : White Rabbit=>  166A3F2A 3FE36E61 , WR/10=>   23DD31DD33057D6, Readout Time =>   23DD31DD3B94000 
aida10 : White Rabbit=>  166A3F2A 4E83A1F7 , WR/10=>   23DD31DD4A6C365, Readout Time =>   23DD31DD46A8000 
aida11 : White Rabbit=>  166A3F2A 5FD19086 , WR/10=>   23DD31DD661C1A7, Readout Time =>   23DD31DD7664000 
aida12 : White Rabbit=>  166A3F2A 6FE06F3D , WR/10=>   23DD31DD7FCD7EC, Readout Time =>   23DD31DD9684000 

04:07 Beam is back AIDA on file R6_815
      Statistics - attachment 8
      FEE Temperatures - attachment 9
      Bias and leakage currents ok - attachment 10
      ASIC check ok

      Multiple merge data error events observed

06:04 All system wide checks ok *except aida06 fails clock status 6*
      Statistics - attachment 11
      FEE Temperatures - attachment 12
      Bias and leakage currents ok - attachment 13

07:22 Beam noticed dropped but back quickly
Attachment 1: 210308_0014_Stats.png
210308_0014_Stats.png
Attachment 2: 2303108_0015_Temp.png
2303108_0015_Temp.png
Attachment 3: 210308_0016_Bias.png
210308_0016_Bias.png
Attachment 4: 210308_0120_System.png
210308_0120_System.png
Attachment 5: 210308_0213_Stats.png
210308_0213_Stats.png
Attachment 6: 210308_0213_Temp.png
210308_0213_Temp.png
Attachment 7: 210308_0215_Bias.png
210308_0215_Bias.png
Attachment 8: 210308_0403_Stats.png
210308_0403_Stats.png
Attachment 9: 210308_0404_Temp.png
210308_0404_Temp.png
Attachment 10: 210308_04011_Bias.png
210308_04011_Bias.png
Attachment 11: 210308_0602_Stats.png
210308_0602_Stats.png
Attachment 12: 210308_0603_Temp.png
210308_0603_Temp.png
Attachment 13: 210308_0604_Bias.png
210308_0604_Bias.png
  177   Sun Mar 7 07:23:26 2021 CA, LSMarch 7th
08:00 ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


08:30 - all system wide checks ok

      - Temperatures ok - attachment 1

      - good event statistics ok - attachment 2

      - detector bias/leakage currents ok - attachment 3

      DAQ continues to run - R1_1148

      Merger ok - 4.6M data items/s

      TapeServer ok - 43MB/s

08:55 Giovanna Benzoni - confirms contribution of light ions - attachment 4 see AOQ plots

09:19 zeroed all histograms

      writing to file R1_1204

      FRS on file 72

      DESPEC to add AIDA XY hit patterns to online analysis histograms

09:35 beam stop

09:56 beam back 

      writing to file R1_1249

      rate spectra - attachment 5

09:30 AIDA DAQ crashed - lost connection to aida05

      power cycle and reset

09:40 AIDA ok now, writing to file R2

09:50 LS takes over

11.10 (German time)
      System wide checks produces some fails: 
      FEE64 module aida06 global clocks failed, 6 (screenshot6)
      This error is seen in RIKEN too, tried a RESYNC from master timestamp tab but error still there

      FEE64 module aida06 failed
      FEE64 module aida07 failed
      Calibration test result: Passed 10, Failed 2 (screenshot7)
      Tried a recalibration for the two FEEs in FADC align & control tab but error still there

      Rest of the checks pass
      Statistics (screenshot8)
      FEE temps okay (screenshot9)
      Leakage currents written to sheets (screenshot10)
      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s

13.00 System wide checks produces same fails as last time: 
      FEE64 module aida06 global clocks failed, 6
      FEE64 module aida06 failed
      FEE64 module aida07 failed
      Calibration test result: Passed 10, Failed 2
      Rest of the checks are okay

      Statistics (screenshot11)
      Spectra rate (screenshot12)
      FEE temps okay (screenshot13)
      Leakage currents written to sheets (screenshot14)
      Merger ~4.5M data items/s
      TapeServer ~ 45MB/s
      There is 1.5TB left on current drive

14.00 beam stopped to look at bPlast, have use the time to change slow comparator threshold
      Slow comparators in FEE9-12 raised to 0xd
      

      

      
      
      



        
Attachment 1: Screenshot_from_2021-03-07_07-30-28.png
Screenshot_from_2021-03-07_07-30-28.png
Attachment 2: Screenshot_from_2021-03-07_07-31-27.png
Screenshot_from_2021-03-07_07-31-27.png
Attachment 3: Screenshot_from_2021-03-07_07-31-55.png
Screenshot_from_2021-03-07_07-31-55.png
Attachment 4: Screenshot_from_2021-03-07_07-53-05.png
Screenshot_from_2021-03-07_07-53-05.png
Attachment 5: Screenshot_from_2021-03-07_08-59-23.png
Screenshot_from_2021-03-07_08-59-23.png
Attachment 6: 1110_Systemchecks.PNG
1110_Systemchecks.PNG
Attachment 7: 1110_Systemchecks2.PNG
1110_Systemchecks2.PNG
Attachment 8: 1110_Statistics.PNG
1110_Statistics.PNG
Attachment 9: 1110_FEEtemps.PNG
1110_FEEtemps.PNG
Attachment 10: 1110_Leakagecurrents.PNG
1110_Leakagecurrents.PNG
Attachment 11: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 12: 1300_Spectrumrate.PNG
1300_Spectrumrate.PNG
Attachment 13: 1300_FEEtemps.PNG
1300_FEEtemps.PNG
Attachment 14: 1300_Leakagecurrents.PNG
1300_Leakagecurrents.PNG
  176   Sat Mar 6 23:37:22 2021 TDSunday 7 March 00.00-08:00
00.30 UNILAC problem - no beam

00:39 DAQ continues R1_518
      FRS 190W setting 

      ASIC settinfs 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xa

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

01:07 all system wide checks OK

      detector biases & leakage currents OK - see attachment 1
      FEE64 temperatures OK - see attachment 2
      good event statistics - OK - see attachment 3
      Merger OK - see attachment 4
      TapeServer OK - see attachment 5

01:20 zero'd all histograms
      ASIC check load

      rate spectra - see attachment 6

01:48 beam return

      run74 (R1_608)

03.05 all system wide checks OK

      detector biases & leakage currents OK - see attachment 7
      FEE64 temperatures OK - see attachment 8
      good event statistics - OK - see attachment 9
      Merger OK - see attachment 10
      TapeServer OK - see attachment 11

     rate spectra - see attachment 12

04.46 run74 ends (R1_841)

05.00 run75 (R1_859)

05:40 detector biases & leakage currents OK - see attachment 13
      FEE64 temperatures OK - see attachment 14
      good event statistics - OK - see attachment 15
      Merger OK - see attachment 16
      TapeServer OK - see attachment 17

     rate spectra - see attachment 18

07:29 run76 (R1_1055)
Attachment 1: 1.png
1.png
Attachment 2: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 3: Screenshot_2021-03-07_Statistics_aidas-gsi.png
Screenshot_2021-03-07_Statistics_aidas-gsi.png
Attachment 4: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi.png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi.png
Attachment 5: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(1).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(1).png
Attachment 6: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(2).png
Attachment 7: 2.png
2.png
Attachment 8: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 9: Screenshot_2021-03-07_Statistics_aidas-gsi(1).png
Screenshot_2021-03-07_Statistics_aidas-gsi(1).png
Attachment 10: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(1).png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(1).png
Attachment 11: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(2).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(2).png
Attachment 12: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(3).png
Attachment 13: 3.png
3.png
Attachment 14: Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-03-07_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 15: Screenshot_2021-03-07_Statistics_aidas-gsi(2).png
Screenshot_2021-03-07_Statistics_aidas-gsi(2).png
Attachment 16: Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(2).png
Screenshot_2021-03-07_NewMerger_Control_aidas-gsi(2).png
Attachment 17: Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(3).png
Screenshot_2021-03-07_Tape_Service_(Expert)_aidas-gsi(3).png
Attachment 18: Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-03-07_Spectrum_Browser_aidas-gsi(4).png
  175   Sat Mar 6 06:52:06 2021 OH, LSSaturday 6th March
07:50 Pulser settings - attachment 1

07:52 System wide checks all ok
      Statistics ok - attachment 2
      Bias ok - attachment 3

09:30 The intensity of fragments is being increased
      They are trying to do a factor of 10 increase first which should take us to around 100Hz

10:00 System wide checks all ok
      Statistics - attachment 4
      Layout 1 showing implants - attachment 5
      Fee temperatures ok - attachment 6

12.00 System wide checks all okay, no failures
      Statistics (screenshot7)
      Spectrum rate (screenshot8)
      FEE temps were normal except AIDA04 had "no response" (screenshot9), reloaded several minutes later seems to read normal 
      (screenshot10)
      Leakage currents okay, increase from previous values written to sheets (screenshot12)
      
      Rechecked FEETemps this time AIDA03 had "no response" (screenshot11), seems this occurs when the reload takes longer 
than 
      usual, another reload shows all FEE temps as normal.
      
      Merger and tape server okay

14.00 System wide checks all okay, no failures
      Statistics (screenshot13)
      Spectrum rate (screenshot14)
      FEE temps normal, no issue with reload this time (screenshot15)
      Leakage currents okay still increasing, written to sheets (screenshot16)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

14.50 Beam has been shifted to centre beam more onto AIDA (corresponds to run S452f011)

16.00 System wide checks all okay, no failures
      Statistics (screenshot17)


      Spectrum rate (screenshot18)      
      FEE temps normal except AIDA08 "no response" (screenshot19), reload returns to normal (screenshot20)
      Leakage currents added to sheet (screenshot21)
      Merger okay ~4.5M items/sec
      Tape server ~4.5Mb/sec

17.17 no beam, preparing for the 190Ta setting

17:44 OH Takes over
      System wide checks all ok
      Statistics - attachment 22
      Temp ok - Attachment 23
      Bias and leakage currents ok - Attatchment 24

      While MBS was not currently writing data the merger -> tapeserver link was briefly toggle
      The Directory for the experiment was then setup S452
      The merger -> Tapeserver link was then re-enabled and data forwarding to MBS has continued.

      ASIC Check all ok
      

18:10 Started writing data to file
      File directory S452
      Run number R1_0
      Seems to have skipped a few files First full file R1_10
      Currently writing to /media/SecondDrive/TapeData
      Current free space 4211084180 kB
      Current write speed 46163 kB/s
      Time remaining until full 91222 seconds
      25 hours remaining space

18:34 DESPEC starting new MBS file
      AIDA Currently on file R1_41

19:00 DESPEC Closing file
      AIDA Currently on R1_75

19:05 Comparing last years statistics to this years

FEE	Last year	This year    Factor increase
1	60960	        183111       3.00
2	183494	        201256       1.09
3	91506	        196647       2.15
4	73914	        208290       2.81
5	31979	        63571        1.99
6	54084	        144638       2.67
7	35621	        137605       3.86
8	38626	        95858        2.48
9	264436	        225790       0.85
10	209709	        198896       0.94
11	88443	        133899       1.51
12	183493	        204996       1.12

Sum     1316265	        1994557      1.51

19:22 DESPEC opens a new file (They forgot to mention they were closing the previous file)
      AIDA on R1_104

      Looking int to high write  rate to see if we can do anything to reduce it
      It isn't the correlation scaler as that is coming in at a rate of 16kHz which amounts to around 1.5MB/s - attachment 24

19:46 Much of the rate is coming from a small number of channels - attachment 25

19:17 WR Timestamp error.
      Mentioned to Nic and he said that sometimes small glitches like this are observed
		 Base 		Current 	Difference
aida01 fault 	 0x9e8b : 	 0x9e8d : 	 2  
aida02 fault 	 0x9a8f : 	 0x9a91 : 	 2  
aida03 fault 	 0x29e9 : 	 0x29eb : 	 2  
aida04 fault 	 0x7f66 : 	 0x7f68 : 	 2  
aida05 fault 	 0x1f4f : 	 0x1f51 : 	 2  
aida06 fault 	 0x8faa : 	 0x8fac : 	 2  
aida07 fault 	 0x53f0 : 	 0x53f2 : 	 2  
aida08 fault 	 0x7066 : 	 0x7068 : 	 2  
White Rabbit error counter test result: Passed 4, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     All other checks passed

20:52 DESPEC Stopping file
      AIDA on R1_221


20:53 DESPEC Stopped file because UCESB crashed
      AIDA on file R1_237

21:20 All system wide  checks ok
      Statistics DISC info- Attachment 27
      FEE Temp ok - Attachment 28
      Bias and leakage current ok - Attachment 29
      Statistics good events - Attachment 30

20:51 Started compressing data with command nice -n 10 gzip -v R1_*
      Have discussed with Helena and Nic about the potential issues we could run into with the high data rate
      I am not sure if MBS can keep up with this rate
      We are still writing to file though.

23:00 Correlations have been lost with the FRS DAQ and the rest of the DESPEC analysis
23:10 UCESB restarted and correlations have been regained. Verified with the implant-FRS time difference peak at 13us
Attachment 1: Pulser.png
Pulser.png
Attachment 2: 210306_0752_Stats.png
210306_0752_Stats.png
Attachment 3: 210306_0754_Bias.png
210306_0754_Bias.png
Attachment 4: 210306_0958_Stats.png
210306_0958_Stats.png
Attachment 5: 210306_0959_Layout1.png
210306_0959_Layout1.png
Attachment 6: 210306_1000_Temp.png
210306_1000_Temp.png
Attachment 7: 1102_Statistics.PNG
1102_Statistics.PNG
Attachment 8: 1102_SpectrumRate.PNG
1102_SpectrumRate.PNG
Attachment 9: 1102_FEETemps.PNG
1102_FEETemps.PNG
Attachment 10: 1102_FEETemps2.PNG
1102_FEETemps2.PNG
Attachment 11: 1102_FEETemps3.PNG
1102_FEETemps3.PNG
Attachment 12: 1102_LeakageCurrents.PNG
1102_LeakageCurrents.PNG
Attachment 13: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 14: 1300_SpectrumRate.PNG
1300_SpectrumRate.PNG
Attachment 15: 1300_FEETemps.PNG
1300_FEETemps.PNG
Attachment 16: 1300_LeakageCurrents.PNG
1300_LeakageCurrents.PNG
Attachment 17: 1500_Statistics.PNG
1500_Statistics.PNG
Attachment 18: 1500_SpectrumRate.PNG
1500_SpectrumRate.PNG
Attachment 19: 1500_FEETemps.PNG
1500_FEETemps.PNG
Attachment 20: 1500_FEETemps2.PNG
1500_FEETemps2.PNG
Attachment 21: 1500_LeakageCurrents.PNG
1500_LeakageCurrents.PNG
Attachment 22: 210306_1726_Stats.png
210306_1726_Stats.png
Attachment 23: 210306_1747_Temp.png
210306_1747_Temp.png
Attachment 24: 210327_1748_Bias.png
210327_1748_Bias.png
Attachment 25: 210326_1920_MergerStats.png
210326_1920_MergerStats.png
Attachment 26: 210306_1946_Layout1.png
210306_1946_Layout1.png
Attachment 27: 210306_2120_Stats.png
210306_2120_Stats.png
Attachment 28: 210306_2121_Temp.png
210306_2121_Temp.png
Attachment 29: 210306_2123_Bias.png
210306_2123_Bias.png
Attachment 30: 210326_1920_MergerStats.png
210326_1920_MergerStats.png
  174   Fri Mar 5 23:45:41 2021 TDS452 information
aida-gsi Anydesk 897170655

DESPEC remote working - https://sf.gsi.de/d/a4fb2134e06a450ca777/

S452 shift checks - attachment 1

S452 shift plan - PDF attachment 2 - Google doc https://docs.google.com/spreadsheets/d/1hjNM5xdPoxq0MfNE5ILnNrHm2QKW7Pgp1UKF7p70ghQ/edit#gid=1745844766

Mattermost - https://mattermost.gsi.de/despec/channels/experiment-s452-nearline-sort

DESPEC Zoom meeting

Join Zoom Meeting
https://gsi-fair.zoom.us/j/98980822569

Meeting ID: 989 8082 2569
Passcode: S452

FRS Zoom meeting 

Meeting-ID: 973 4809 5114

password: FRS-BT2021

Kenncode: 456657
Attachment 1: S452__Shifts_checkpoints_v1_(1).pdf
S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf S452__Shifts_checkpoints_v1_(1).pdf
Attachment 2: S452_Shift_Plan.pdf
S452_Shift_Plan.pdf
  173   Fri Mar 5 23:21:59 2021 CA, OHMarch 6th 2021 00:00 - 08:00 shift
00:23 206Hg setting - No beam in AIDA yet

      AIDA not writing to file

      Data being forwarded to MBS

      Fragments expected tonight (?)

00:45 GSI DAQ restart

      FEE64 Temps ok - attachment 1

      All system wide checks ok

01:00 Good event statistics - attachment 2

01:48 detector biases/ leakage currents - ok - attachment 3

02:10 beam on, plug removed

      beam delivered to AIDA

      rates spectra - attachment 4

      peaks at ~ 10 counts/channel in DSSD1 c.2000Hz

02:20 beam stop

02:26 beam restart - rate spectra - attachment 5

02:43 high energy channel spectra - attachment 6

      saved the high energy spectra to layout 2 in spectrum browser

03:00 beam stop

03:08 temperatures ok - attachment 7

      statistics ok - attachment 8

      bias/leakage currents ok - attachment 9

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

      all system wide checks ok

04:24 beam is back

      rate spectra - attachment 10

04:51 206Hg implanting in DSSD2

      Rate spectra - attachment 11

      HEC spectra - attachment 12

05:08 temperatures ok - attachment 13

      statistics ok - attachment 14

      detector bias/leakage currents ok - attachment 15

      All system wide checks ok

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok 

05:45 beam stopped

05:50 beam back

06:45 beam running, but no implants in AIDA

      possible issues with beam delivery to S4

      NewMerger ok - ~4.5M data items merge rate

      Tape service ok - ~46MB/s, NoStorage

      Data forwarding to MBS ok

07:10 temperatures ok - attachment 16

      statistics ok - attachment 17

      detector bias/leakage currents ok - attachment 18

      all system wide checks OK














      

      

      
Attachment 1: Screenshot_from_2021-03-06_00-00-27.png
Screenshot_from_2021-03-06_00-00-27.png
Attachment 2: Screenshot_from_2021-03-06_00-05-18.png
Screenshot_from_2021-03-06_00-05-18.png
Attachment 3: Screenshot_from_2021-03-06_00-47-41.png
Screenshot_from_2021-03-06_00-47-41.png
Attachment 4: Screenshot_from_2021-03-06_01-17-12.png
Screenshot_from_2021-03-06_01-17-12.png
Attachment 5: Screenshot_from_2021-03-06_01-25-51.png
Screenshot_from_2021-03-06_01-25-51.png
Attachment 6: Screenshot_from_2021-03-06_01-42-58.png
Screenshot_from_2021-03-06_01-42-58.png
Attachment 7: Screenshot_from_2021-03-06_02-05-46.png
Screenshot_from_2021-03-06_02-05-46.png
Attachment 8: Screenshot_from_2021-03-06_02-06-32.png
Screenshot_from_2021-03-06_02-06-32.png
Attachment 9: Screenshot_from_2021-03-06_02-07-47.png
Screenshot_from_2021-03-06_02-07-47.png
Attachment 10: Screenshot_from_2021-03-06_03-24-28.png
Screenshot_from_2021-03-06_03-24-28.png
Attachment 11: Screenshot_from_2021-03-06_03-53-35.png
Screenshot_from_2021-03-06_03-53-35.png
Attachment 12: Screenshot_from_2021-03-06_03-54-17_-_1.png
Screenshot_from_2021-03-06_03-54-17_-_1.png
Attachment 13: Screenshot_from_2021-03-06_04-09-25.png
Screenshot_from_2021-03-06_04-09-25.png
Attachment 14: Screenshot_from_2021-03-06_04-10-07.png
Screenshot_from_2021-03-06_04-10-07.png
Attachment 15: Screenshot_from_2021-03-06_04-10-32.png
Screenshot_from_2021-03-06_04-10-32.png
Attachment 16: Screenshot_from_2021-03-06_06-07-19.png
Screenshot_from_2021-03-06_06-07-19.png
Attachment 17: Screenshot_from_2021-03-06_06-08-16.png
Screenshot_from_2021-03-06_06-08-16.png
Attachment 18: Screenshot_from_2021-03-06_06-08-44.png
Screenshot_from_2021-03-06_06-08-44.png
  172   Fri Mar 5 22:53:05 2021 OHThursday 5th of March
There has been no beam for much of the day. It was lost at 13:00 and not regained until 22:00 German time.
At some point during the day the White Rabbit timestamp cable was unplugged from AIDA. This caused it lose correlation with the rest of the system.
At first a reset and setup was performed to try and regain correlation but that did not prove successful.
A full powercycle was then performed at ~22:30 German time which upon restarting the DAQ solved the issue.

The plan is to overnight run some fragments, for the purposes of observing the isomers. This data is not essential and as such we will not be writing to disk.

Beam plug is in front of AIDA. This is causing a huge spike in the Ge.
  171   Thu Mar 4 11:16:13 2021 OHFurther cases of possible database issues
During testing for the S452 experiment yesterday we were running through the steps of a restart.

The FEEs were not powercycled at this stage but the MIDAS server was restarted.
Initial setup went smoothly and no issues were encountered during setup.
NewMerger and TapeServer were both setup and set to going.
Upon ''Going'' the run control the data transfer for aida09 dropped out.
Going to the NewMerger it could be seen that data was making it through and data items were being merged.
The tapeserver however was seeing no data and neither was the MBS dataspy.
During this time the FEEs became unresponsive likely as their buffers filled. Trying to reset gave the attached errors.

Checking the folder manually it could be seen that the Options file was there.

At this point a powercycle was pewrformed but upong 'Going' the DAQ the same issue was again encountered.

This time we were able to recover control access to the FEEs by running multiple NewMerger sessions which helped clear the buffers.

During our search for the problem we looked at the Options from within MIDAS and could not see anything out of the ordinary.
We also tried restoring the Options from within MIDAS.
We again reached the same issue with the tapeserver.

At this point we restored the Options folder from by copying manually within terminal.
Following this we were able to start the DAQ normally.
Attachment 1: ErrorMessage.PNG
ErrorMessage.PNG
Attachment 2: errorMessage.txt
EXPERIMENTS/AIDA/Options/aida04 TS_SYNC_PHASE NOT available: error 0x10004; value TS_SYNC_PHASE does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ExtClk NOT available: error 0x10004; value ExtClk does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.shift NOT available: error 0x10004; value Aida.shift does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 MACB_TRIG_MODE NOT available: error 0x10004; value MACB_TRIG_MODE does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.offset NOT available: error 0x10004; value Aida.offset does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 RunNumber NOT available: error 0x10004; value RunNumber does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_GroupBase NOT available: error 0x10004; value Aida_GroupBase does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Rate.channels NOT available: error 0x10004; value Rate.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Stat.channels NOT available: error 0x10004; value Stat.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.Vchannels NOT available: error 0x10004; value Aida.Vchannels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ASIC.settings NOT available: error 0x10004; value ASIC.settings does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_D_Enable NOT available: error 0x10004; value Aida_Hist_D_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 WAVE_DMA_HWM NOT available: error 0x10004; value WAVE_DMA_HWM does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Stat.shift NOT available: error 0x10004; value Stat.shift does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.Wchannels NOT available: error 0x10004; value Aida.Wchannels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_V_Enable NOT available: error 0x10004; value Aida_Hist_V_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Include.Aida NOT available: error 0x10004; value Include.Aida does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.channels NOT available: error 0x10004; value Aida.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 DataAcqPgm NOT available: error 0x10004; value DataAcqPgm does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_H_Enable NOT available: error 0x10004; value Aida_Hist_H_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 DataFormat NOT available: error 0x10004; value DataFormat does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ASIC_DMA_HWM NOT available: error 0x10004; value ASIC_DMA_HWM does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_L_Enable NOT available: error 0x10004; value Aida_Hist_L_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
STATE for aida01 returned with an error
error: SOAP http transport timed out after 20000 ms
NONE
error: SOAP http transport timed out after 20000 ms
    while executing
"$transport $procVarName $url $req"
    (procedure "::SOAP::invoke" line 18)
    invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
    ("eval" body line 1)
    invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
    (procedure "DataAcquisitionControlClient__GetState" line 1)
    invoked from within
"DataAcquisitionControlClient__GetState"
  170   Thu Mar 4 11:03:04 2021 NHS452 Operating Notes
Please note the following "quirks" currently present in S452 that are important

a) Strips/Thresholds
Thresholds are 0xA, 0xA, 0x10 [Maybe changed depending on rate, etc]
Some strips are very noisy - damage?

b) Powering FEE64s
Due to a misbehaving USB hub(?) or cable only 7 FEE TTYs are monitored
aida04 and aida05 can take minutes to start, unsure why.
aida05 is monitored on nnrpi1 and hence when aida05 is ready it is safe to assume all 12 are ready
Watch the logs as normal

c) Slow control & nnrpi2
For the Grafana script to read the DSSD HV, it must be ran in a screen session
If nnrpi2 is restarted please run the HV as a screen by Putty SSHing (localhost 22)
and running
screen -S caenhv /dev/ttyACM0
if screen was just closed you can do
screen -x caenhv
to reconnect

The pulser ideally should be started the same
screen -S pulser /dev/ttyUSB0
To allow remote changing by NH/GSI
The pulser is sent to all subsystems for correlation and should be kept on at 2 Hz for the experiment
IN the pulser putty you can check the settings by typing
display settings
And general commands:
help

The NIM crate can be monitored /dev/ttyUSB2 but you should be careful
It is very easy to turn off the NIM crate if you do
I run it in a tmux session with (
tmux attach -r
) to have it "readonly"

There is a small script on nnrpi2
./check_usbs.sh
to see which ttyUSB is which device
As they can change if the pi is rebooted

d) Scaler Readout
If powercycled FEEs, enable the scaler in Local Controls page by setting BuTiS interface control to 0x3

To enable scaler readout after GOing the run please open the Correlation Control page
(Act on all FEE64s) and set control to 0x1
This has to be done every time the FEE64s are set to GOing

e) New Tape Server
For waveform readout in future experiments, the tape server is in "Expert" mode which is harder to use
Please read the "How to use" instructions by clicking the button on the bottom right
Or in a terminal
cat /MIDAS/TclHttpd/Html/ETape/1.txt

f) GSI/MBS
If GSI team ask to restart AIDA MBS you may need to restart the MBS relay so it can reconnect

To start MBS in a terminal
ssh despec@x86l-94
cd mbsrun/s452/aida_to_mbs
mbs
@startup
It should report it connected to AIDA-3 - if not check the MBS relay is running or restart the relay

Please check the MBS rate command
rate -nst -rst -nev -rev -ndata -rdata -st
That the "Streams" column does not frequently (ever) have 0 empty streams
If so AIDA data may be getting dropped and should be looked into
If another DESPEC DAQ has failed this will eventually happen and is not a problem
But if it occurs in normal operation it is concerning

Do NOT stop the AIDA run without talking to GSI team - if no AIDA data the whole MBS DAQ stops
and blocks all other subsystems. It is best to keep AIDA ticking over constantly and let
MBS start/stop files when relevant.
Of course if things have to stop it is OK but inform GSI shifters so they can take action
  169   Thu Mar 4 10:13:48 2021 OHAnyDesk connection instuction
Attachment 1: Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf
Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf
  168   Wed Mar 3 15:02:19 2021 CAMBS command line startup
/MIDAS/Linux/startup/datarelaymbs 
  167   Sat Feb 27 12:47:57 2021 NHAIDA Rates/Situation
Updated figures as of 27 Feb 2021 - S4 is now closed for experiments but may open on Wednesday.
Snout has been grounded and all FEEs are seemingly grounded properly.

Figure  1: Good Events
Figure  2: Rate Histograms (Log-y)
Figure  3: Waveforms (Odd)
Figure  4: Waveforms (Even)
Figure  5: Waveforms @ 5MS/s (Odd)
Figure  6: Waveforms @ 5MS/s (Even)
Figure  7: Waveforms with Pulser (Odd)
Figure  8: Waveforms with Pulser (Even)*
Figure  9: Waveforms with Pulser @ 5MS/s (Odd)
Figure 10: Waveforms with Pulser @ 5MS/s (Even)*

* y-Scale enlarged from 0-10000 to 0-15000 to capture top of pulse
Attachment 1: firefox_2021-02-27_13-47-19.png
firefox_2021-02-27_13-47-19.png
Attachment 2: firefox_2021-02-27_13-46-59.png
firefox_2021-02-27_13-46-59.png
Attachment 3: firefox_2021-02-27_13-36-27.png
firefox_2021-02-27_13-36-27.png
Attachment 4: firefox_2021-02-27_13-36-52.png
firefox_2021-02-27_13-36-52.png
Attachment 5: firefox_2021-02-27_13-38-19.png
firefox_2021-02-27_13-38-19.png
Attachment 6: firefox_2021-02-27_13-37-53.png
firefox_2021-02-27_13-37-53.png
Attachment 7: firefox_2021-02-27_13-39-48.png
firefox_2021-02-27_13-39-48.png
Attachment 8: firefox_2021-02-27_13-42-55.png
firefox_2021-02-27_13-42-55.png
Attachment 9: firefox_2021-02-27_13-39-33.png
firefox_2021-02-27_13-39-33.png
Attachment 10: firefox_2021-02-27_13-44-10.png
firefox_2021-02-27_13-44-10.png
  166   Mon Feb 22 18:18:47 2021 NHAIDA Rates/Situation
The ground cable (LEMO00) in aida07 was disconnected, it has been reinserted and the waveforms now look good.

(All data below taken with thresholds 0xA for all)

Noise in aida09 is visible and aida12 is quite high too. (All of DSSD3? Except aida10?)
Event rate in them is very high. The lower frequency noise visible in figs 5,7 seems slow enough for the ASICs
DSSD#2 has nice event rate and DSSD#1 seems "OK"

Leakage currents high but S4 very warm at the moment as roof is still closed.

Some PCBs from bPlas are touching the ribbon cables, so should be insulated better.

Next S4 access later this week will reseat all connectors to ensure good connection.
Other ideas can also be done

Figures: 1, 2 - Good Event + ADC Data Stats
Figure 3 - Rate Spectra (log scale)
Figure 4, 5 - Odd FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 6, 7 - Even FEE64s Waveforms, 50 MS/s (Sample rate 0) and 5 MS/s (Sample rate 10)
Figure 8 - Grafana (DSSD HV)

aida01-aida04 DSSD 1
aida05-aida08 DSSD 2
aida09-aida12 DSSD 3
Attachment 1: GoodRates.png
GoodRates.png
Attachment 2: ADCRates.png
ADCRates.png
Attachment 3: Rates.png
Rates.png
Attachment 4: Waves_0.png
Waves_0.png
Attachment 5: Waves_10.png
Waves_10.png
Attachment 6: EvenWaves_0.png
EvenWaves_0.png
Attachment 7: EvenWaves_10.png
EvenWaves_10.png
Attachment 8: Grafana.png
Grafana.png
  165   Fri Feb 12 17:31:04 2021 NHBackground run
A background run has been started to check the AIDA channels.
It is in /TapeData/BGFEB21

Due to some noisy channels the data rate is high (1.5 MB/s) even at 0x64

S4 access will be limited. 

I do not know when open if it is worth looking at changing the adapter PCBs of it is more likely DSSD internal.
  164   Sat Feb 6 10:45:55 2021 NHAIDA Status Feb 2021
> 01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
> A new 7 port USB hub has also been placed for the last few serial connections.
> 
> For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)
> 
> The system has been powered up and biased successfully. To be monitored over the next few days.
> 
> A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
> The rack will also be grounded properly on Wednesday to investigate the impact on the noise.
> 
> A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...
> 
> The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.
> 
> Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently

05.02.2021 -
The serial hub was disconnected and was reconnected. Unfortunately nnrpi1 seems to have gone offline so cannot be used still. Will try to reboot pi when possible.

The hose has arrived and the water line will be prepared in my office mid next week.

All the leaf MACBs have been updated to the latest firmware (MACB_GSI_Simple.jed) and set to mode '0'. A fourth leaf MACB is now in the NIM crate but has no HDMI connections in or out yet.
All 12 FEEs powered up and have no WR issues (one MABC was first set to wrong mode, but was fixed)

A new tape server to support unmerged waveform readout has been loaded and documented by VP. This will largely be unused but the option is there and will be tested.

The rack was grounded to the platform floor with a large cable to potentially improve situation. Waveforms before (fig 1-2) and after (fig 3-4) attached.

There will be a DESPEC Dry Run next week Mon-Tue.
Attachment 1: pregrd050222_E.png
pregrd050222_E.png
Attachment 2: pregrd050221_O.png
pregrd050221_O.png
Attachment 3: postgrd050221_O.png
postgrd050221_O.png
Attachment 4: postgrd050221_E.png
postgrd050221_E.png
  163   Mon Feb 1 15:19:07 2021 NHAIDA Status Feb 2021
01.02.2021 - Four new sets of PSU, HDMI & Ethernet cables were ran from the DESPEC rack to the AIDA frame, in preparation for 4 new FEE64s.
A new 7 port USB hub has also been placed for the last few serial connections.

For the first experiment this year, we will run with a narrow AIDA configuration as in 2020. (12 FEEs)

The system has been powered up and biased successfully. To be monitored over the next few days.

A lot of the FEE serial monitors are missing, I think a hub has been disconnected from nnrpi1. This will be investigated on Wednesday.
The rack will also be grounded properly on Wednesday to investigate the impact on the noise.

A single new water line (feed and return) needs to be installed for the 16th FEE, this will be done once the hose is found/ordered...

The 4 new FEE64s will be mounted and powered up at some point to obtain the MAC addresses, and possibly check MIDAS merging 16 FEE64s.

Waveform data will be taken wednesday before & after grounding the rack to see noise situation currently
  162   Tue Nov 24 13:10:42 2020 NHWaveforms with frame grounded
AIDA Frame is now grounded to platform (will add photo later)

Waveforms improved from April (c.f. https://elog.ph.ed.ac.uk/DESPEC/125) but comparison not truly fair (other systems may be offline now)
Attachment 1: waves_odd.png
waves_odd.png
Attachment 2: waves_even.png
waves_even.png
Attachment 3: 20201124_164416.jpg
20201124_164416.jpg
Attachment 4: 20201124_164413.jpg
20201124_164413.jpg
  161   Tue Oct 6 15:48:06 2020 NHnnrpi kernel update
FYI:

The SD card on nnrpi2 was full again causing issues, the log files in /var/log (kern.log & messages) were *huge* and filled with the same error many times a second

Oct  4 15:01:22 raspberrypi kernel: [1030048.603846]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.607742] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.607742]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.667755] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.667755]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.667848] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.667848]                                                                                                                                                                                                                                                                         
Oct  4 15:01:22 raspberrypi kernel: [1030048.731753] WARN::dwc_otg_handle_mode_mismatch_intr:68: Mode Mismatch Interrupt: currently in Host mode                                                                                                                                                                             
Oct  4 15:01:22 raspberrypi kernel: [1030048.731753]                                                                                             

A google issue showed that this was fixed in an update to Linux kernel, so I updated the kernel to the latest version.
now the message doesn't seem to appear so hopefully the pis won't break so rapidly.
  160   Tue Jul 7 08:50:27 2020 NHnnrpi2 does not boot
MIDAS was not working properly on nnrpi2 - power cycled. Now system does not boot properly, it gets stuck trying to login claiming "PAM critical error" - probable file system corruption?
Will investigate if SD card can be checked and repaired
  159   Tue Jul 7 08:04:57 2020 NHPower failure in Messhuette
Last week there was a power failure in Messhuette (FRS control room) - AIDA-3 and MBS were affected.
AIDA itself and S4 equipment is on UPS and was not shut down.

AIDA-3 has been restarted and MIDAS restarted. All seems OK

S4 conditions: 24.2 C / 33.5% RH / Td = 7.1 C 
  157   Wed Mar 18 18:02:37 2020 PJCSNew Firmware loaded in all FEE64s

Loaded new frimware into all 12 FEE64s in the system. IWR_Dec19_4.bin 

Booted ok after power-cycle.  

RESET/SETUP/GO all fine.

system wide checks ( SWC ) clocks, white rabbit ( WR ) OK, ADC calibrated : aida10 failed. Couldn't get them to calibrate.

Checked aida01 waveforms for ASIC1. set threshold for LED to 5000 to stop triggering in the noise.

Set White Rabbit register SYNC Rollover Target ( 14 ) to 0x4000 instead of 0xe to reduce the data rate to better understand the waveform activity.

All FEE64s are set to enable all channels of ASIC1 with low thresholds so a lot of data moving. Checked activity using statistics window.

Checked waveforms working for ASIC1 of aida01. Noisy but capturing the pulser pulse.

 Enabled all the waveforms on aida01 to see if they all function ok. After a few seconds waveform readout stopped. Not sure of the reason.

STOP, change all four thresholds to 5000, Check ASIC controls : all ok, GO. No activity.

RESET/SETUP ok.

SWC : clock failed, ADC failed , WR ok ????

Power-cycle...... concludes ok, RESET/SETUP ok, SWC : clocks all failed, ADC all failed, WR ok.

Checked WR status browser page for aida01. WR timestamp 0 ! status : 0xA should be 0 .

Checked tcl for SWC WR check and it doesn't check the status. Changed at DL and at GSI and now it does.

so SWC WR check no all fails.

Contacted NH and he found Fatima VME crate with WR source in had been switched off. It was powered on again.

RESET/SETUP , SWC: clocks all ok, ADC all ok, WR all ok.

set aida01 and all odd number  LED thresholds to 5000 and aida02 all even number thresholds to 10000 and activity of statistic Wave Good Events is similar across the system. 

Checked each of the Waveform capture controls pages ( only shows the status registers for ASIC1 ) and all are fine, nothing sticking.

enabled all the ASIC waveforms on aida01 again and it sticks, clear out three ASICs and operate the waveform reset in the System Functions and it restarts and runs.

So a qualified success. Don't enable too many channels at once ?

left running at 18:00 UK time and checked at 21:30 UK time , still operating ok.

 

 

  156   Mon Mar 16 07:25:24 2020 NHEnd of Experiment

08:25 Beam Stopped, end of S480

Tape Server stopped at R12_2097

Started again in no storage mode

Will leave compressing runs

  155   Mon Mar 16 05:08:12 2020 TA, AM, ESRegular entry
02:14 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at around 3M items/sec
      tape service running at 26MB/sec

 

All attached to this entry with screenshots to 1-7

Attachment 1: 16__1.png
16__1.png
Attachment 2: 16__2.png
16__2.png
Attachment 3: 16__3.png
16__3.png
Attachment 4: 16__4.png
16__4.png
Attachment 5: 16__5.png
16__5.png
Attachment 6: 16__6.png
16__6.png
  154   Mon Mar 16 01:07:29 2020 TA, AM, ESRegular entry
02:14 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at 3M items/sec
      tape service running at 26MB/sec

 

All attached to this entry with screenshots to 1-7

 

 

04:05 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at 3M items/sec
      tape service running at 26MB/sec

 

All attached tp this entry with screenshots 7-14

Attachment 1: Screenshot-2.png
Screenshot-2.png
Attachment 2: Screenshot-4.png
Screenshot-4.png
Attachment 3: Screenshot-3.png
Screenshot-3.png
Attachment 4: Screenshot-5.png
Screenshot-5.png
Attachment 5: Screenshot-7.png
Screenshot-7.png
Attachment 6: Screenshot.png
Screenshot.png
Attachment 7: 16_1.png
16_1.png
Attachment 8: 16_2.png
16_2.png
Attachment 9: 16_3.png
16_3.png
Attachment 10: 16_4.png
16_4.png
Attachment 11: 16_5.png
16_5.png
Attachment 12: 16_6.png
16_6.png
Attachment 13: 16_7.png
16_7.png
  153   Sun Mar 15 06:45:00 2020 CA, LS, CGSunday 15th March 08:00 - 24:00
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

07:46 all system wide checks ok

      good event statistics ok - attachment 1
  
      FEE64 temperatures ok - attachment 2

      detector bias/leakage currents ok - attachment 3

08:48 merger ok ~ 3 million data items/s

      TapeServer ok ~ 27 MB/s

      data forwarding to MBS ok

08:50 AIDA writing to file SecondDrive/TapeData/S480/R12_1524

09:56 all system wide checks ok

      good event statistics ok - attachment 4

      detector bias/leakage currents ok - attachment 5
  
      FEE64 temperatures ok - attachment 6

12:09 all system wide checks ok


12:13 all system wide checks ok

      good event statistics ok - attachment 7
  
      FEE64 temperatures ok - attachment 8

      detector bias/leakage currents ok, but ramping up - attachment 9


13:32 merger ok ~ 3 million data items/s

      TapeServer ok ~ 23 MB/s

      data forwarding to MBS ok

      Beam seems to be off

13:33 AIDA writing to file R12_1743

*update* checked DESPEC S480 Elog, issues with UNILAC, beam lost at 13:29

13:55 beam back

13:57 AIDA writing to file R12_1759

14:35 all system wide checks ok

      FEE64 temperatures ok - attachment 10

      good event statistics ok - attachment 11
  
      detector bias/leakage currents ok, but ramping up - attachment 12

15:18 merger ok ~ 2.7 million data items/s

      TapeServer ok ~ 27 MB/s

      data forwarding to MBS ok

      AIDA writing to file R12_1283

16:00 1.8.H high energy spectra - attachments 13 & 14

      hit-rate spectra - attachment 15
 
16.22 all system wide checks okay 
      FEE temperatures okay (attachment 16)
      leakage currents okay and recorded to spreadsheet (attachment 17)
      good event stats okay (attachment 18)
      merger running at 3M items/sec
      tape service running at 26MB/sec

18.00 began compressing R7 files with command: nice -n 10 gzip -v R7_* &
      
      all system wide checks okay 
      FEE temperatures okay (attachment 19)
      leakage currents okay and recorded to spreadsheet (attachment 20)
      good event stats okay (attachment 21)
      merger running at 3M items/sec
      tape service running at 27MB/sec    

20.10 all system wide checks okay 
      FEE temperatures okay (attachment 22)
      leakage currents okay and recorded to spreadsheet, detectors 1 and 2 still rising (attachment 23)
      good event stats okay (attachment 24)
      merger running at 3M items/sec
      tape service running at 25MB/sec

      rates histogram (attachment 25)
      low energy histograms (attachment 26 and 27)
      high energy histograms (attachments 28 and 29)

      
22.07 all system wide checks okay 
      FEE temperatures okay (attachment 30)
      leakage currents okay and recorded to spreadsheet (attachment 31)
      good event stats okay (attachment 32)
      merger running at 3M items/sec
      tape service running at 25MB/sec
      currently on R12_2097

23.56 all system wide checks okay 
      FEE temperatures okay (attachment 33)
      leakage currents okay and recorded to spreadsheet (attachment 34)
      good event stats okay (attachment 35)
      merger running at 3M items/sec
      tape service running at 26MB/sec
      
finishing up now. night shift group have been informed how to check up on AIDA, there is a sheet on the table with relevant information
      
Attachment 1: 200315_0749_stats.png
200315_0749_stats.png
Attachment 2: 200315_0749_temp.png
200315_0749_temp.png
Attachment 3: 200315_0750_bias.png
200315_0750_bias.png
Attachment 4: 200315_0956_stats.png
200315_0956_stats.png
Attachment 5: 200315_0957_bias.png
200315_0957_bias.png
Attachment 6: 200315_0957_temp.png
200315_0957_temp.png
Attachment 7: 200315_1210_stats.png
200315_1210_stats.png
Attachment 8: 200315_1211_temp.png
200315_1211_temp.png
Attachment 9: 200315_1212_bias.png
200315_1212_bias.png
Attachment 10: corrigan_temperature.png
corrigan_temperature.png
Attachment 11: corrigan_stats.png
corrigan_stats.png
Attachment 12: corrigan_bias.png
corrigan_bias.png
Attachment 13: corriganhe1.png
corriganhe1.png
Attachment 14: corriganhe2.png
corriganhe2.png
Attachment 15: 200315_1553_rates.png
200315_1553_rates.png
Attachment 16: 200315_1619_feetemps.png
200315_1619_feetemps.png
Attachment 17: 200315_1620_leakagecurrents.png
200315_1620_leakagecurrents.png
Attachment 18: 200315_1620_goodevents.png
200315_1620_goodevents.png
Attachment 19: 200315_1758_feetemps.png
200315_1758_feetemps.png
Attachment 20: 200315_1759_leakagecurrents.png
200315_1759_leakagecurrents.png
Attachment 21: 200315_1758_goodevents.png
200315_1758_goodevents.png
Attachment 22: 200315_2004_feetemps.png
200315_2004_feetemps.png
Attachment 23: 200315_2007_leakagecurrents#.png
200315_2007_leakagecurrents#.png
Attachment 24: 200315_2006_goodevents.png
200315_2006_goodevents.png
Attachment 25: 200315_2009_rates.png
200315_2009_rates.png
Attachment 26: 200315_2010_lowodd.png
200315_2010_lowodd.png
Attachment 27: 200315_2011_loweven.png
200315_2011_loweven.png
Attachment 28: 200315_2012_highodd.png
200315_2012_highodd.png
Attachment 29: 200315_2014_higheven.png
200315_2014_higheven.png
Attachment 30: 200315_2202_feetemps.png
200315_2202_feetemps.png
Attachment 31: 200315_2203_leakagecurrents.png
200315_2203_leakagecurrents.png
Attachment 32: 200315_2204_goodevents.png
200315_2204_goodevents.png
Attachment 33: 200315_2358_FEETEMPS.png
200315_2358_FEETEMPS.png
Attachment 34: 200315_2358LEAKAGECURRENTS.png
200315_2358LEAKAGECURRENTS.png
Attachment 35: 200315_2359_goodevents.png
200315_2359_goodevents.png
  152   Sat Mar 14 23:09:28 2020 MB, SJ, MRSunday 15th March 0:00-8:00

AIDA ELog filled out by DESPEC night shift members Matthew Brunet (MB), Shaheen Jazrawi (SJ), and Matthias Rudiger (MR).

0:05-0:08 Beam operators altering settings, beam off/sporadic during this period.

0:46
    All system wide checks passed
    Leak currents lower than previous measurement - attachment #1
    Implants look as previously - attachment #2
    Stats about where they have been - attachment #3
    FEE temperatures fine - attachment #4
    Merger fine, 3 Mega items / sec
    TapeServer writing about 25 MB/s

 

2:46

      All system wide checks passed
     Leak currents similar to previous - attachment #5 
     Implants look as previously - attachment #6
     Stats about where they have been - attachment #7
     FEE temperatures fine - attachment #8
     Merger fine, 3 Mega items / sec
     TapeServer writing about 27 MB/s
 

4:49

      All system wide checks passed
     Leak currents similar to previous - attachment #9
     Implants look as previously - attachment #10
     Stats good - attachment #11
     FEE temperatures fine - attachment #12
     Merger fine, 3 Mega items / sec
     TapeServer writing about 27 MB/s

6:49

      All system wide checks passed
     Leak currents no significant change - attachment #13
     Implants look as previously - attachment #14
     Stats good - attachment #15
     FEE temperatures fine - attachment #16
     Merger fine, 3 Mega items / sec
     TapeServer writing about 25 MB/s

Attachment 1: Matt-15-BiasesA.png
Matt-15-BiasesA.png
Attachment 2: Matt-15-HistosA.png
Matt-15-HistosA.png
Attachment 3: Matt-15-StatsA.png
Matt-15-StatsA.png
Attachment 4: Matt-15-TempsA-.png
Matt-15-TempsA-.png
Attachment 5: Matt-15-BiasesB.png
Matt-15-BiasesB.png
Attachment 6: Matt-15-HistosB.png
Matt-15-HistosB.png
Attachment 7: Matt-15-StatsB.png
Matt-15-StatsB.png
Attachment 8: Matt-15-TempsB.png
Matt-15-TempsB.png
Attachment 9: Matt-15-BiasesC.png
Matt-15-BiasesC.png
Attachment 10: Matt-15-HistosC.png
Matt-15-HistosC.png
Attachment 11: Matt-15-StatsC.png
Matt-15-StatsC.png
Attachment 12: Matt-15-TempsC.png
Matt-15-TempsC.png
Attachment 13: Matt-15-BiasesD.png
Matt-15-BiasesD.png
Attachment 14: Matt-15-HistosD.png
Matt-15-HistosD.png
Attachment 15: Matt-15-StatsD.png
Matt-15-StatsD.png
Attachment 16: Matt-15-TempsD.png
Matt-15-TempsD.png
  151   Sat Mar 14 07:13:59 2020 DK, TD, LS, CGSaturday 14th March 08:00 - 24:00
8:15
    All system wide checks passed
    Leak currents are continuing to creep up - attachment #1
    FEE temperatures fine - attachment #2
    Stats about where they have been - attachment #3
    Implants look as previously (5ish counts per channel in SSD1 and SSD2) - attachment #4
    Merger fine, 3 Mega items / sec
    TapeServer still writing about 25 MB/s

09.37 analysis of data file S480/R12_443
      deadtime aida09 c. 5%, all other FEE64s <<1%
      See attachment #5

09.54.51
      biases, SSD1 and 2 over 2 uA each now -  attachment #6

11:20
    All  system-wide checks okay
    Merger okay, 3 Mega items / sec
    TapeServer normal, 27 MB/sec
    Biases as attachment #7
    Stats as #8
    Temps as #9
    Histos as #10

14:10
    All system wide checks okay
    Stats #11
    Temps #12
    Implants #13
    Leak currents are getting a bit higher #14

15:30
    All system wide checks passed
    Stats attachment  #15
    Temps attachment  #16
    Implants  attachment #17
    Biases / leak currents attachment  #18
    Merger okay, 3 Mega items / sec
    TapeServer okay, writing 27 MB/s

16.12 all online spectra zero'd

16.31 stat & rate spectra - attachments 19-22
      1.8.L spectra - attachments 23 & 24
      aida03 pulser peak 57 ch FWHM
      1.8.H spectra - attachments 25 & 26
      NewMerger & TapeServer - attachments 27 & 28

18.29

     system wide checks all okay
     FEE temperatures normal (attachment 29)
     Leakage currents okay and recorded to spreadsheet (attachment 30)
     Good event statistics okay (attachment 31)
     Merger running at 3M items/sec
     Tape server running at 25MB/sec

20.25
     system wide checks all okay
     FEE temperatures normal (attachment 32)
     Leakage currents okay and recorded to spreadsheet (attachment 33)
     Good event statistics okay (attachment 34)
     Merger running at 3M items/sec
     Tape server running at 26MB/sec

as of 20.39 just finished R12_961

21.36 beam is down confirmed by rates histogram (attachment 35), informed others on shift
      was down for maybe 5 minutes is back on at 21.40 (attachment 36)
      think between R12_1004 to R12_1008

22.10
     system wide checks all okay
     FEE temperatures normal (attachment 37)
     Leakage currents okay and recorded to spreadsheet (attachment 38)
     Good event statistics okay (attachment 39)
     Merger running at 3M items/sec
     Tape server running at 26MB/sec

23.03 someone comes to ask if we see an increase in implantation rate as they do their side
      I checked the rates histogram and good events and see nothing that would indicate an increase (attachment 40 and 41)
      FEE temperatures and leakage currents do not seem any different either
      believe they are calling someone about the issue

23.30 leaving now to catch bus but people on night shift have been told how to check on AIDA and know that CG is on call if needed
Attachment 1: daid-14-biasA.png
daid-14-biasA.png
Attachment 2: daid-14-tempA.png
daid-14-tempA.png
Attachment 3: daid-14-statsA.png
daid-14-statsA.png
Attachment 4: daid-14-histosA.png
daid-14-histosA.png
Attachment 5: R12_443
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   86997080 ( 1109430.4 Hz)
        Other data format:  174986804 ( 2231519.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         29 (       0.4 Hz)
                               RESUME:         29 (       0.4 Hz)
                              SYNC100:   87327942 ( 1113649.7 Hz)
                              WR48-63:   87327942 ( 1113649.7 Hz)
                           FEE64 disc:     330862 (    4219.3 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     102097 (    1302.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       78.416 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.116                0.000
  3                0.000                0.000
  4                0.000                0.000
  5                0.066                0.000
  6                0.000                0.000
  7                0.000                0.000
  8                4.659                3.647
  9                0.000                0.000
 10                0.112                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    3143962    6596287          0          0          1          1    3246749    3246749     102787          0          0      19287
  1   11441532   22943556          0          0          0          0   11461696   11461696      20164          0          0       9627
  2    6435721   13042134          0          0          2          2    6492617    6492617      56896          0          0      12957
  3    3633801    7382655          0          0          0          0    3672152    3672152      38351          0          0      19271
  4    1100939    2276428          0          0          0          0    1125789    1125789      24850          0          0       7339
  5    2632637    5296610          0          0          2          2    2643081    2643081      10444          0          0       4759
  6    1846149    3780603          0          0          0          0    1875584    1875584      29435          0          0      12550
  7    2374464    4801965          0          0          0          0    2392143    2392143      17679          0          0       8964
  8   20263627   40557223          0          0         22         22   20273602   20273602       9975          0          0       2005
  9   14853979   29722517          0          0          0          0   14858832   14858832       4853          0          0       1283
 10    6417731   12865067          0          0          2          2    6427598    6427598       9867          0          0       2254
 11   12852538   25721759          0          0          0          0   12858099   12858099       5561          0          0       1801
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:34518.602s (    0.927 blocks/s,   0.058 Mb/s)
Attachment 6: bias-daid-14B.png
bias-daid-14B.png
Attachment 7: daid-14-biasC.png
daid-14-biasC.png
Attachment 8: daid-14-tempC.png
daid-14-tempC.png
Attachment 9: daid-14-statsC.png
daid-14-statsC.png
Attachment 10: daid-14-histosC.png
daid-14-histosC.png
Attachment 11: daid-14-tempsD.png
daid-14-tempsD.png
Attachment 12: daid-14-statsD.png
daid-14-statsD.png
Attachment 13: daid-14-histosD.png
daid-14-histosD.png
Attachment 14: daid-14-biasD.png
daid-14-biasD.png
Attachment 15: daid-14-statsE.png
daid-14-statsE.png
Attachment 16: daid-14-tempsE.png
daid-14-tempsE.png
Attachment 17: daid-14-histosE.png
daid-14-histosE.png
Attachment 18: daid-14-biasE.png
daid-14-biasE.png
Attachment 19: 1.png
1.png
Attachment 20: 2.png
2.png
Attachment 21: 3.png
3.png
Attachment 22: 4.png
4.png
Attachment 23: 5.png
5.png
Attachment 24: 6.png
6.png
Attachment 25: 7.png
7.png
Attachment 26: 8.png
8.png
Attachment 27: 9.png
9.png
Attachment 28: 10.png
10.png
Attachment 29: 200314_1823_feetemps.png
200314_1823_feetemps.png
Attachment 30: 200314_1827_leakagecurrents.png
200314_1827_leakagecurrents.png
Attachment 31: 200314_1824_goodevents.png
200314_1824_goodevents.png
Attachment 32: 200314_2020_feetemps.png
200314_2020_feetemps.png
Attachment 33: 200314_2023_leakagecurrents.png
200314_2023_leakagecurrents.png
Attachment 34: 200314_2022_goodevents.png
200314_2022_goodevents.png
Attachment 35: 200314_2136_BEAMDOWN.png
200314_2136_BEAMDOWN.png
Attachment 36: 200314_2141_beamback.png
200314_2141_beamback.png
Attachment 37: 200314_2207_feetemps.png
200314_2207_feetemps.png
Attachment 38: 200314_2210_leakagecurrents.png
200314_2210_leakagecurrents.png
Attachment 39: 200314_2208_goodevents.png
200314_2208_goodevents.png
Attachment 40: 200314_2303_rates.png
200314_2303_rates.png
Attachment 41: 200314_2303_goodevents.png
200314_2303_goodevents.png
  150   Fri Mar 13 23:06:39 2020 CA, OHSaturday 14th March 00:00 - 08:00
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

00.00 DAQ momentarily set to NoStorage
      OH changes symbolic link such that AIDA is writing directly to mounted hard-drives ("SecondDrive" and "ThirdDrive")

00.02 DAQ now writing to file again - /media/SecondDrive/TapeData/S480/R12_0

00.30 all system wide checks ok

      FEE64 temperatures ok - attachment 1

      good event statistics ok - attachment 2

      detector bias and leakage currents ok - attachment 3
 
00:43 merger ok - writing at ~ 3 million items/s

      TapeServer ok - writing at ~ 27 Mb/s

      Data forwarding to MBS ok

02:14 all system wide checks ok

      FEE64 temperatures ok - attachment 4

      good event statistics ok - attachment 5

      detector bias and leakage currents ok - attachment 6

02:17 Merger ok  ~ 2.7 million items/s

      TapeServer ok  ~ 27 Mb/s

      Data forwarding to MBS ok

02:18 currently writing to file /media/SecondDrive/TapeData/S480/R12_110

02:57 attachments 7 & 8 - high energy channel spectra

      attachment 9 - hit rate spectra

04:21  FEE64 temperatures ok - attachment 10

      good event statistics ok - attachment 11

      detector bias and leakage currents ok - attachment 12

      system wide checks ok - except WR decoder status (see attachment 13)

04:26 check ASIC control

04:30 system functions -> select required function -> collect all WhiteRabbit status error counters for baseline
      WR decoder status check now ok

04:50 Merger ok  ~ 3 million items/s

      TapeServer ok  ~ 25 Mb/s

      Data forwarding to MBS ok

04:52 currently writing to file /media/SecondDrive/TapeData/S480/R12_231
 
06:11 all system wide checks ok

      FEE64 temperatures ok - attachment 14

      good event statistics ok - attachment 15

      detector bias and leakage currents ok - attachment 16


06:41 Merger ok  ~ 3.3 million items/s

      TapeServer ok  ~ 25 Mb/s

      Data forwarding to MBS ok

06:42 currently writing to file /media/SecondDrive/TapeData/S480/R12_315

07:25 hit-rate spectra - attachment 17

07:36 low energy channel spectra (1.8.L) - attachments 18 & 19



 


      
Attachment 1: 200314_0020_temp.png
200314_0020_temp.png
Attachment 2: 200314_0030_stats.png
200314_0030_stats.png
Attachment 3: 200314_0031_bias.png
200314_0031_bias.png
Attachment 4: 200314_0212_temp.png
200314_0212_temp.png
Attachment 5: 200314_0213_stats.png
200314_0213_stats.png
Attachment 6: 200314_0214_bias.png
200314_0214_bias.png
Attachment 7: even_he.png
even_he.png
Attachment 8: odd_he.png
odd_he.png
Attachment 9: 200314_0254_rates.png
200314_0254_rates.png
Attachment 10: 200314_0416_temp.png
200314_0416_temp.png
Attachment 11: 200314_0417_stats.png
200314_0417_stats.png
Attachment 12: 200314_0418_bias.png
200314_0418_bias.png
Attachment 13: 200314_WRerror.png
200314_WRerror.png
Attachment 14: 200314_0608_temp.png
200314_0608_temp.png
Attachment 15: 200314_0609_stats.png
200314_0609_stats.png
Attachment 16: 200314_0610_bias.png
200314_0610_bias.png
Attachment 17: 200314_0724_rates.png
200314_0724_rates.png
Attachment 18: 200314_0733_18l_odd.png
200314_0733_18l_odd.png
Attachment 19: 200314_0735_18l_even.png
200314_0735_18l_even.png
  149   Fri Mar 13 15:48:11 2020 Friday 13th 16:00 - 24:00OH, TD
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

16:51 Bias and leakage currents ok but starting to increase in DSSD 1 predominantly - attachment 1
      Statistics ok - attachment 2
      FEE temp ok - attachment 3
      System wide checks all ok 
     
16:57 Histograms zeroed

17:17 ASIC control check done

17:40 Layout 1 - attachment 4
      Layouts 3 and 4 - attachments 4 and 5
      Layouts 5 and 6 - attachments 6 and 7

19:06 Earlier in the shift two extra 8TB drives were formatted and mounted
      Named SecondDrive and ThirdDrive
      TapeData folder created on SecondDrive
      Plan is before the end of this shift to move the symbolic link from the filling up drive to the new drive
      Current prediction is drive would otherwise be full by 12:20 tomorrow.
      Playing it safe and switching early

21:57 Bias and leakage ok - attachment 8
      Statistics ok - attachment 9
      FEE Temps ok - attachment 10
Attachment 1: 200313_1649_bias.png
200313_1649_bias.png
Attachment 2: 200313_1652_stats.png
200313_1652_stats.png
Attachment 3: 200313_1653_temp.png
200313_1653_temp.png
Attachment 4: 200313_1738_Layout1.png
200313_1738_Layout1.png
Attachment 5: 200313_1739_layout2.png
200313_1739_layout2.png
Attachment 6: 200313_1740_layout4.png
200313_1740_layout4.png
Attachment 7: 200313_1740_layout5.png
200313_1740_layout5.png
Attachment 8: 200313_1741_layout6.png
200313_1741_layout6.png
Attachment 9: 200313_2155_bias
Attachment 10: 200313_2156_stats.png
200313_2156_stats.png
Attachment 11: 200313_2156_Temps.png
200313_2156_Temps.png
  148   Fri Mar 13 08:01:53 2020 DKFriday 13th 8:00 to 16:00 shift
8:00 - See previous shift's elog entry.  All good.

9:00
    All system wide checks passed
    Merger okay, 3 Mega items / sec, all nnaida twinkle or stay lit
    Tape Server okay - 27 MB/s
    Histograms of implants look normal, about 5 counts per bin in SSD1 and SSD2 - see attachment #1
    Stats as usual - attachment #2
    Temps normal - attachment #3
    Biases and leaks fine -  attachment #4

Disk usage projection
    Space remaining: ~2 TB
    Fill rate: ~2 GB / 1 minute
    -> 1000 minutes =~ 17 hours

Still gzipping from R6_823

10:13 I noticed that the integer for the TapeServer UI feedbackk has been exceeded.
     	Kbytes Written:  -1986382400	Kbytes/sec:  26816
        Rate is fixed at that value on update.
        However, actual data files are being updated so data has been coming.
        DK and NH consulted and believe the situation is fine.
        We agreed next time the DESPEC team closes a run and starts a new one, they inform DK so he can restart the TapeServer for posterity and peace of mind.

10:25 Now we are on R10 after start and stop the TapeServer.  Problem is resolved.

11:30
      Implants: attachment #5
      Stats: attachment #6
      Biases / leaks: attachment #7
      Temps: attachment #8
      System wide checks performed, situation normal.
Attachment 1: daid-13-histosA.png
daid-13-histosA.png
Attachment 2: daid-13-statsA.png
daid-13-statsA.png
Attachment 3: daid-13-tempsA.png
daid-13-tempsA.png
Attachment 4: daid-13-biasA.png
daid-13-biasA.png
Attachment 5: daid-13-implantsB.png
daid-13-implantsB.png
Attachment 6: daid-13-statsB.png
daid-13-statsB.png
Attachment 7: daid-13-biasesB.png
daid-13-biasesB.png
Attachment 8: daid-13-tempsB.png
daid-13-tempsB.png
  147   Thu Mar 12 23:44:48 2020 DKPID Plots
Labelled PID plots.  Note that the A/Q calibration is shifted by +0.02 units, and Z calibration is offset by about +5.

There was an error in the printouts for Cd isotopes, but the below attachments are corrected.

The Z-depth (color, not nuclear charge) is log scale.
Attachment 1: pid.png
pid.png
Attachment 2: pid.png
pid.png
Attachment 3: pid.pdf
pid.pdf
  146   Thu Mar 12 23:04:24 2020 LS, DKFriday 13th 00:00 to 08:00 shift
   Shift file starting R9_639

   system wide checks okay
   FEE temperatures okay (attachment 1)
   leakage currents normal and recorded to spreadsheet (attachment 2)
   good event stats are normal (attachment 3)
   merger running at 3.5 M items/sec
   tape running at 26 MB/sec
   
   rates histogram (attachment 4)
   low energy histograms (attachment 5 and 6)
   high energy histograms (attachment 7 and 8)

02.02 
   system wide checks okay
   FEE temperatures okay (attachment 9)
   leakage currents normal and recorded to spreadsheet (attachment 10)
   good event stats are normal (attachment 11)
   merger running at 3.5 M items/sec
   tape running at 27 MB/sec

04.09 
   system wide checks okay
   FEE temperatures okay (attachment 12)
   leakage currents normal and recorded to spreadsheet (attachment 13)
   good event stats are normal (attachment 14)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
   writing to file R9_840

06.07 
   system wide checks okay
   FEE temperatures okay (attachment 15)
   leakage currents normal and recorded to spreadsheet (attachment 16)
   good event stats are normal (attachment 17)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
 
07.59 
   system wide checks okay
   FEE temperatures okay (attachment 18)
   leakage currents normal and recorded to spreadsheet (attachment 19)
   good event stats are normal (attachment 20)
   merger running at 3 M items/sec
   tape running at 26 MB/sec

as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space
Attachment 1: 200313_0006_feetemps.png
200313_0006_feetemps.png
Attachment 2: 200313_0008_leakcurrents.png
200313_0008_leakcurrents.png
Attachment 3: 200313_0008_goodevents.png
200313_0008_goodevents.png
Attachment 4: 200313_0015_rateshisto.png
200313_0015_rateshisto.png
Attachment 5: 200313_0018_loweoddfees.png
200313_0018_loweoddfees.png
Attachment 6: 200313_0019_loweevenfees.png
200313_0019_loweevenfees.png
Attachment 7: 200313_0021_higheoddfees.png
200313_0021_higheoddfees.png
Attachment 8: 200313_0022_higheevenfees.png
200313_0022_higheevenfees.png
Attachment 9: 200313_0203_feetemps.png
200313_0203_feetemps.png
Attachment 10: 200313_0203_leakagecurrents.png
200313_0203_leakagecurrents.png
Attachment 11: 200313_0205_goodevents.png
200313_0205_goodevents.png
Attachment 12: 200313_0403_feetemps.png
200313_0403_feetemps.png
Attachment 13: 200313_0408_leakagecurrents.png
200313_0408_leakagecurrents.png
Attachment 14: 200313_0406_goodevents.png
200313_0406_goodevents.png
Attachment 15: 200313_0602_feetemps.png
200313_0602_feetemps.png
Attachment 16: 200313_0605_leakagecurrents.png
200313_0605_leakagecurrents.png
Attachment 17: 200313_0605_goodevents.png
200313_0605_goodevents.png
Attachment 18: 200313_0756_feetemps.png
200313_0756_feetemps.png
Attachment 19: 200313_0758_leakagecurrents.png
200313_0758_leakagecurrents.png
Attachment 20: 200313_0758_goodevents.png
200313_0758_goodevents.png
  145   Thu Mar 12 15:25:08 2020 Thursday, March 12 16:00 to midnight shiftCA, DK, LS
16:30
    All system wide checks passed
    Temps shown in attachment #1 
    Histos shown in attachment #2
      NB: Zoomed in, linear scale
    Biases shown in attachment #3
    Stats shown in attachment #4
    Merger rates fine 3 Mega items / sec
    TapeServer looks happy, 25 MB/s

Paddy was asking us to give a rough estimate for the number of implants per spill.
I zoomed in on the rates histogram (attachment #2) and it looks like roughly 5 implants per channel in SSD 1 and SSD2.
So this is 5*128*2=1280.  Note that I don't know the rate window size (e.g., 1 second, 1 spill?)  Probably the window does not catch a full spill.
This might be an over-estimate because an implant in SSD2 may leave a signal in SSD1.
Paddy was getting around 1k or 2k fragments per spill, but to first order it seems consistent.

18:36
   Near R9_372 they slightly opened some FRS slits, as there is a belief we may be blocking some of 94Pd.
   Implant rate seems similar to before, see attachment #5
   Biases did not change more than normal, see attachment #6

20:15
   All system wide checks passed
   Temps shown in attachment #7
   Stats shown in attachment #8
   Biases shown in attchment #9

22:12 
   All system wide checks okay
   Histo #10
   Bias #11
   Stat #12
   Temps #13
Attachment 1: daid-12-tempA.png
daid-12-tempA.png
Attachment 2: daid-12-histosA.png
daid-12-histosA.png
Attachment 3: daid-12-biasesA.png
daid-12-biasesA.png
Attachment 4: daid-12-histosB.png
daid-12-histosB.png
Attachment 5: daid-12-biasB.png
daid-12-biasB.png
Attachment 6: daid-12-tempC.png
daid-12-tempC.png
Attachment 7: daid-12-statsC.png
daid-12-statsC.png
Attachment 8: daid-12-biasC.png
daid-12-biasC.png
Attachment 9: daid-12-histosD.png
daid-12-histosD.png
Attachment 10: daid-12-biasesD.png
daid-12-biasesD.png
Attachment 11: daid-12-statsD.png
daid-12-statsD.png
Attachment 12: daid-12-tempsD.png
daid-12-tempsD.png
  144   Thu Mar 12 08:04:24 2020 OH, CA, TDThursday 12th March 08:00-16:00
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

09:00 Bias and leakage currents ok - attachment 1
      Statistics ok - attachment 2
      FEE temperatures ok - attachment 3
      System wide checks all ok

09:08 No beam, not sure exactly how long it has been down.
      FRS team are working on it

09:28 still no beam. issues with SIS. 
      DAQ set to NoStorage mode
      most recent file R7_522

10.27 beam is back
      writing to file R9_0

10.43 attachment 4 - hit rate spectra

11:09 all system wide checks ok
      FEE64 temps ok - attachment 5
      good event statistics ok - attachment 6
      detector bias and leakage currents - attachment 7

11.18 merger ok, ~ 2.7 million data items per second
      TapeServer ok, ~ 25 Mb/s data rate
      Data forwarding to MBS ok

11.20 writing to file R9_39

11.22 beam off

11.24 beam back online (R9_42)

11.41 FRS has reverted to an older setting

13:04 hit rate spectra - attachment 8

13:19 all system wide checks ok

13:30 FEE64 temps ok - attachment 9

      good event statistics ok - attachment 10

      detector bias and leakage currents ok - attachment 11


15:31 good event statistics ok - attachment 13

      detector bias and leakage currents ok - attachment 14

      FEE64 temps ok - attachment 15

      all system wide checks ok

      beam seems to have stopped during 2pm meeting

15.35 TapeServer ok, data rate ~ 23 Mb/s

      Merger ok, rate of 2.5 - 3 million data items per second

      data forwarding to MBS ok

15.38 AIDA writing to file R9_233

16:02 beam back - AIDA writing to file R9_250







      
Attachment 1: 200312_0902_bias.png
200312_0902_bias.png
Attachment 2: 200312_0902_stats.png
200312_0902_stats.png
Attachment 3: 200312_0903_temp.png
200312_0903_temp.png
Attachment 4: 200312_1042_rates.png
200312_1042_rates.png
Attachment 5: 200312_1105_temp.png
200312_1105_temp.png
Attachment 6: 200312_1107_stats.png
200312_1107_stats.png
Attachment 7: 200312_1108_stats.png
200312_1108_stats.png
Attachment 8: 200312_1302_rates.png
200312_1302_rates.png
Attachment 9: 200312_1321_temp.png
200312_1321_temp.png
Attachment 10: 200312_1325_stats.png
200312_1325_stats.png
Attachment 11: 200312_1327_bias.png
200312_1327_bias.png
Attachment 12: 200312_1528_stats.png
200312_1528_stats.png
Attachment 13: 200312_1529_bias.png
200312_1529_bias.png
Attachment 14: 300312_1527_temp.png
300312_1527_temp.png
  143   Wed Mar 11 23:44:27 2020 OHThursday 12th March 00:00-08:000
ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

00:44 Implantation range in AIDA - attachment 1

01:14 At current rate will run out of space in 35 hours ~ 18:00 hours on Friday 13th March.
      During the day if possible it might be wise to try and obtain a USB3 HDD mount to mount one of the spare drives in
      the AIDA boxes and begin moving files onto that

01:35 System wide checks all ok
      Bias and leakage current ok - attachment 2
      Stats all ok - attachment 3
      FEE temperatures ok - attachment 4

      TapeServer rate around 25-29MB per second
      Merger rate around 2.6-3.2 million items per second

04:11 System wide checks all ok
      Bias and leakage currents ok - attachment 5
      Stats all ok - attachment 6
      FEE temp ok - attachment 7
06:38 System wide checks all ok
      Bias and leakage currents ok - attachment 8
      Stats all ok - attachment 9
      FEE temp ok - attachment 10
Attachment 1: 200311_ImpDis.png
200311_ImpDis.png
Attachment 2: 200312_0135_Bias.png
200312_0135_Bias.png
Attachment 3: 200312_0136_Stats.png
200312_0136_Stats.png
Attachment 4: 200312_0139_FEETemp.png
200312_0139_FEETemp.png
Attachment 5: 200312_0405_BIAS.png
200312_0405_BIAS.png
Attachment 6: 200312_0405_stats.png
200312_0405_stats.png
Attachment 7: 200312_0406_Temp.png
200312_0406_Temp.png
Attachment 8: 200312_0638_bias.png
200312_0638_bias.png
Attachment 9: 200312_0639_stats.png
200312_0639_stats.png
Attachment 10: 200312_0639_temp.png
200312_0639_temp.png
  142   Wed Mar 11 17:21:33 2020 CA, TD, OHWednesday 11 March 16.00-00.00
18.25 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - OK

WR timestamp control - attachments 4 & 5

merger & TapeServer OK - attachments 6-8

rate spectra - attachment 9

1.8.L spectra - attachments 10 & 11
  aida03 72 ch FWHM

20:36 DSSSD biases & leakage currents OK - attachment 12
      FEE64 temps OK - attachment 13
      good events statistics OK - attachment 14

      all system wide checks ok

21.50 beam starts back up
      DAQ running, TapeServer now writing to file R7_0
      data rate ~ 25 Mb/s
      data forwarding to MBS ok
      merger ok, data item rate ~ 2.7 million items/sec
      spill time length is now 3 seconds
      1500 count rate per spill, ~ 300 Hz implant rate in AIDA

22.35 FEE64 temps OK - attachment 15
      good event statistics ok - attachment 16
      detector bias/ leakage currents ok - attachment 17

      all system wide checks ok

22.47 attachments 18 & 19 - high energy spectra for all FEE64 (1.8.H)

      attachment 20 - hit rate spectra for all FEE64

23.02 beam down
      on file R7_55

23.11 beam back online
      on file R7_61

23.15 system wide checks all ok

      detector bias/ leakage currents ok - attachment 21

      good event statistics ok - attachment 22

      FEE64 temperatures ok - attachment 23      
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 10.png
10.png
Attachment 10: 11.png
11.png
Attachment 11: 12.png
12.png
Attachment 12: 200311_2035_bias.png
200311_2035_bias.png
Attachment 13: 200311_2034_temps.png
200311_2034_temps.png
Attachment 14: 200311_2034_stats.png
200311_2034_stats.png
Attachment 15: 200311_2233_temp.png
200311_2233_temp.png
Attachment 16: 200311_2234_stats.png
200311_2234_stats.png
Attachment 17: 200311_2235_bias.png
200311_2235_bias.png
Attachment 18: 200311_18h_odd.png
200311_18h_odd.png
Attachment 19: 200311_2245_18h_even.png
200311_2245_18h_even.png
Attachment 20: 200311_2242_rates.png
200311_2242_rates.png
Attachment 21: 200311_2314_bias.png
200311_2314_bias.png
Attachment 22: 200311_2314_stats.png
200311_2314_stats.png
Attachment 23: 201103_2313_temp.png
201103_2313_temp.png
  141   Wed Mar 11 16:29:17 2020 CA, LS, TDslow comparator threshold comparison
17.29 good event statistics for slow comparator threshold at 100 keV (0xa) - attachment 1

17.36 good event statistics for slow comparator threshold at 110 keV (0xb) - attachment 2

17.39 good event statistics for slow comparator threshold at 120 keV (0xc) - attachment 3

17.53 good event statistics for slow comparator threshold at 80 keV (0x8) for DSSD2, and 100keV (0xa) for DSSD 1&3 - attachment 4
Attachment 1: 100keV_stats.png
100keV_stats.png
Attachment 2: 110keVStats.png
110keVStats.png
Attachment 3: 120kevstats.png
120kevstats.png
Attachment 4: 80kevDSSD2.png
80kevDSSD2.png
  140   Tue Mar 10 23:17:31 2020 DK, LS, PW, TD, CAWednesday 11 March 2020
00:15 Shift change near AIDA R6_475

  We continue with spill cycles around 0.2 Hz or one beginning once each ~5 seconds.

  All system-wide checks passed successfully
  Temperatures ok (aida01 still a bit hot) - see attachment #1
  Stats as normal (SSD3 still noisy) - see attachment #2
  Bias and pulser settings remain sane - see attachment #3
  Writing data to disk at ~26 MB/s as yesterday - see attachment #4
  Merging fine, 2.6 to 3.3 Mega items / sec (all aida## seen to go on) - see attachment #5
  Rates look not unreasonable for expected implantation in short time window - see attachment #6

2:26
   We are near R6_574
   All system checks are fine
   Temps are fine - attachment #7
   Stats look as before - attachment #8
   Biases and leaks typical - attachment #9
   Merger fine, 3 x 10^6 items / sec (skip screencap)
   TapeServer still sits near 25 MB/s (skip screencap)
   Cumulative implants look reasonable, but NB that histos have not been cleared - attachment #10

2:44 R6_587 or R6_588
   The spills seem to have stopped based on the beeping scalars.  Confirmed with other group there are no scint counts.
   See attachment of AIDA rates where no implants can be seen.  See attachment #11

2:46 Spill beeps heard briefly, but they cut again.  R6_590

2:49 Beam seems to be back R6_592
  Scints have counts in DESPEC DAQ, scalars are beeping, AIDA sees implants.  No problem.  See attachment #12 for implants.

4:35 
  System checks all pass (except expected error on master clock)
  Temps okay - attachment #13
  Biases and leak currents normal - attachment #14
  Stats as before - attachment #15
  Merger going, 3 mega items per sec
  TapeServer going, 25 MB/s

5:59 Beam is gone again, near R6_738.  As before, we get consistent confirmation from the trigger clicks and the plastic scint data of DESPEC group.
   See attachment #16 which shows no implants in the rate.


   

Contact phones for Control Room: 2222

We called, they are aware of a problem with the magnets and they are working to resolve it.  So we will await them.

Expert is scheduled for 7:30, but he may come earlier like 7:00 if he can make it in.  

We leave AIDA writing to disk since there are some longer lived daughters (hour or more) so we may still get some interesting decays with beam off

7:14  Beam is still off, but I make some notes
   System wide checks all okay
   Temps as normal (see attachment #17)
   Biases and leak currents okay (see attachment #18)
   Stats as normal (see attachment #19)
   Merger going at 3 mega cps
   TapeServer on, around 25 MB/sec
   Implants not seen in histogram rate as expected (see attachment #20)

7:32 We called the operators to confirm that they don't change anything before calling us first.
     Surely they said they don't touch the settings without telling us first, and just one magnet (maybe upstream of SIS) is down, and no plan to alter settings.

8.00 shift change at file R6_823
     beam is back on, no settings changed.
     particles per spill is similar to before (2k~3k per spill)
     implants now being seen in rate histogram (attachment 21)

8.32 system wide checks all okay
     FEE temperatures are normal (attachment 22)
     leak currents are okay and have been recorded to sheet (attachment 23)
     good event stats seem normal, although noted that all FEEs in DSSD3 now over 100k (attachment 24)
     merger rate is running around 3M items/sec
     tape service rate is running around 26MB/sec

9.20 high energy spectra histograms for even FEEs was not displaying correctly (attachment 25). can also be seen on yesterdays elog on attachment 34
     problem can be seen clearly in /tmp/LayOut6.mlf for the line corresponding to aida4 was 1.8h not 1.8H as the information was somehow saved in the layout (attachment 26).
     remaking each histogram and saving layout6 seems to be fixed (attachment 27)

Top of the error message showed the problem, like:
====
Using /tmp/LayOut6.mlf
restore server=aida02 spectrum=1.8.H to gallery=1 overlap=1
restore server=aida04 spectrum=1.8.h to gallery=2 overlap=1
SpecDetails returned with an error
0x30004
SOAP-ENV:Server-Error detected in method SpecDetails
Invalid spectrum name
    invoked from within
"SpecDetails 1.8.h"
    (in namespace inscope "::urn:SpectrumService" script line 1)
    invoked from within
====

10.05 beam is being stopped to change some thresholds

10.34 system wide checks all okay
      FEE temperatures are normal (attachment 28)
      leak currents are okay and have been recorded to spreadsheet (attachment 29)
      good event stats are normal, as previously mentioned aida11 rate was over 100k this looks to have just been a fluctuation (attachment 30)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24MB/sec
      implants not seen in histogram rate as expected as beam still down (see attachment 30)

10.42 beam still down but taking data, currently file finishing R6_946
      as of now we have 3.2Tb left which should correspond to around 26 hours left until space is full
      have zeroed all histograms so will take screenshots at 11.00

11.06 checked rates histograms, for aida3 some low energy channels in ASICs 1,2, and 3 were missing so performed ASIC control checks

11.25 beam back file S480/R6_975

12.00 analysis file S480/R6_1001 - attachment 33
      FEE aida09 deadtime c. 5% (highest rate good events c. 250k), all other FEE64s << 1%
      all HEC data rate c. 1kHz
      all ADC data rate c. 1.2MHz

12.23 system wide checks all okay
      FEE temperatures are normal (attachment 34)
      leak currents are okay and have been recorded to spreadsheet (attachment 35)
      good event stats are okay (attachment 36)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 26MB/sec

13.00 beam off, unknown reason at the moment, but can confirm from rates histogram (attachment 37)
      seems to be intermittent. file R6_1051
      
13.10 beam back on, no one was called. currently on file R6_1061

13.22 beam off as going to higher intensity beam. file R6_1069

14.42 system wide checks all okay
      FEE temperatures are normal (attachment 38)
      leak currents are okay and have been recorded to spreadsheet (attachment 39)
      good event stats are okay (attachment 40)
      merger is running at a rate of 3M items/sec
      tape service is running at a rate of 24M/sec
14.48 on file R6_1131

16.19 beam stop for 2 hours. TapeServer stopped and restarted - AIDA writing to No Storage
      file R6_1193

16:35 compression of files R6_[200-999] commenced (nice -20)

16:40 system wide checks all okay
      FEE temperatures are normal (attachment 41)
      leak currents are okay and have been recorded to spreadsheet (attachment 42)
      good event stats are okay (attachment 43)
Attachment 1: daid-11-tempsA.png
daid-11-tempsA.png
Attachment 2: daid-11-statsA.png
daid-11-statsA.png
Attachment 3: daid-11-biasA.png
daid-11-biasA.png
Attachment 4: daid-11-TSA.png
daid-11-TSA.png
Attachment 5: daid-11-mergerA.png
daid-11-mergerA.png
Attachment 6: daid-11-implantsA.png
daid-11-implantsA.png
Attachment 7: daid-11-tempB.png
daid-11-tempB.png
Attachment 8: daid-11-statsB.png
daid-11-statsB.png
Attachment 9: daid-11-biasB.png
daid-11-biasB.png
Attachment 10: daid-11-histoB.png
daid-11-histoB.png
Attachment 11: daid-11-histos-beamdrop.png
daid-11-histos-beamdrop.png
Attachment 12: daid-11-beamreturn.png
daid-11-beamreturn.png
Attachment 13: daid-11-tempC.png
daid-11-tempC.png
Attachment 14: daid-11-biasC.png
daid-11-biasC.png
Attachment 15: daid-11-statsC.png
daid-11-statsC.png
Attachment 16: daid-beamstop2.png
daid-beamstop2.png
Attachment 17: daid-11-tempD.png
daid-11-tempD.png
Attachment 18: daid-11-biasD.png
daid-11-biasD.png
Attachment 19: daid-11-statsD.png
daid-11-statsD.png
Attachment 20: daid-11-histosD.png
daid-11-histosD.png
Attachment 21: 200311_804_rateshistograms.png
200311_804_rateshistograms.png
Attachment 22: 200311_28_feetemps.png
200311_28_feetemps.png
Attachment 23: 200311_831_leakcurrents.png
200311_831_leakcurrents.png
Attachment 24: 200311_829_goodevents.png
200311_829_goodevents.png
Attachment 25: 200311_845_highEevenfees.png
200311_845_highEevenfees.png
Attachment 26: daid-11-histos-lowgaineven.png
daid-11-histos-lowgaineven.png
Attachment 27: 200311_919_highEevenfixed.png
200311_919_highEevenfixed.png
Attachment 28: 200311_1030_feetemps.png
200311_1030_feetemps.png
Attachment 29: 200311_1032_leakcurrents.png
200311_1032_leakcurrents.png
Attachment 30: 200311_1031_goodevents.png
200311_1031_goodevents.png
Attachment 31: 200311_1040_ratehistogram.png
200311_1040_ratehistogram.png
Attachment 32: 200311_1107_rates.png
200311_1107_rates.png
Attachment 33: R6_1001
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:   87080536 ( 1162937.9 Hz)
        Other data format:  174903340 ( 2335788.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:         31 (       0.4 Hz)
                               RESUME:         31 (       0.4 Hz)
                              SYNC100:   87327938 ( 1166241.9 Hz)
                              WR48-63:   87327938 ( 1166241.9 Hz)
                           FEE64 disc:     247402 (    3304.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      73144 (     976.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       74.880 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                0.000                0.000
  2                0.000                0.000
  3                0.144                0.000
  4                0.000                0.000
  5                0.033                0.000
  6                0.000                0.000
  7                0.033                0.000
  8                4.160                3.444
  9                0.111                0.000
 10                0.116                0.000
 11                0.111                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    2537909    5321817          0          0          1          1    2619908    2619908      81999          0          0      13572
  1   12374388   24792837          0          0          0          0   12389075   12389075      14687          0          0       7144
  2    5995747   12118373          0          0          0          0    6038040    6038040      42293          0          0       8968
  3    3245929    6571235          0          0          3          3    3272386    3272386      26457          0          0      13205
  4    1018810    2094536          0          0          0          0    1037782    1037782      18972          0          0       5843
  5    3175371    6374045          0          0          1          1    3183138    3183138       7767          0          0       3654
  6    1851746    3764473          0          0          0          0    1872073    1872073      20327          0          0       9037
  7    2503081    5045008          0          0          1          1    2516029    2516029      12948          0          0       6580
  8   19164914   38353829          0          0         20         20   19172901   19172901       7987          0          0       1389
  9   15009332   30028700          0          0          2          2   15012676   15012676       3344          0          0        894
 10    5573107   11166435          0          0          2          2    5579846    5579846       6739          0          0       1608
 11   14630202   29272052          0          0          1          1   14634084   14634084       3882          0          0       1250
 12          0          0          0          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:43185.973s (    0.741 blocks/s,   0.046 Mb/s)
Attachment 34: 200311_1219_feetemps.png
200311_1219_feetemps.png
Attachment 35: 200311_1221_leakcurrents.png
200311_1221_leakcurrents.png
Attachment 36: 200311_1220_goodevents.png
200311_1220_goodevents.png
Attachment 37: 200311_1300_rateshistogram.png
200311_1300_rateshistogram.png
Attachment 38: 200311_1440_FEETEMPS.png
200311_1440_FEETEMPS.png
Attachment 39: 200311_1442_leakcurrents.png
200311_1442_leakcurrents.png
Attachment 40: 200311_1442_GOODEVENTS.png
200311_1442_GOODEVENTS.png
Attachment 41: 200311_1638_temp.png
200311_1638_temp.png
Attachment 42: 200311_1639_bias.png
200311_1639_bias.png
Attachment 43: 200311_1639_stats.png
200311_1639_stats.png
  139   Tue Mar 10 12:36:44 2020 TD[How To] Monitor AIDA
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.

AIDA has multiple, independent data acquisition servers connected by an Ethernet network.

Response times to commands, information updates etc are ~10-60s - please be patient!


Every 4h

1) Desktop 1 - 1x PUTTY terminal sessions

   Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply 

   Addr 00

   Ch 0 = DSSSD # 1
    :
    :
   Ch 2 = DSSSD # 3

   Select the terminal and obtain a screenshot (Fn-Alt-PrtScn) of DSSSD #1 = #3 detector
   bias voltages and leakage currents
   Upload screenshots to Elog (elog.ph.ed.ac.uk/DESPEC)
   Update leakage current spreadsheet (https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023)

   If S4 ambient temperature information is available please add this information to the Elog

   Compare DSSSD leakage currents (and temperatures) with previous observations 

2) Desktop 1 - Firefox web browser - Temperatures tab

   Select 'Reload'

   Check temperatures

       Virtex < 70 deg C
       PSU    < 45 deg C
       ASIC   < 70 deg C


   Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0

   *If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows

   Desktop 3 - Firefox web browser

   From 'Rly16 Control' select 'Switch ALL off'

   and *now* seek expert help


*If* DAQ running also


3) Desktop 1 - Firefox web browser - Run Control tab

   Select 'Update'

   Check state of *all* data acquisition servers is 'going'
   Check histogramming and data transfer #1 are enabled as indicated by 'H W X' tags

4) Desktop 1 - Firefox web browser - Statistics

   Check 'good event' statistics - screenshot to Elog - compare with previous Elog entries

5) Desktop 1 - Firefox web browser - System Wide Checks

   Check all system wide checks - compare results with porevious Elog entries

6) Desktop 2 - Firefox web browser - MERGE tab

   Select 'Reload'

   Check merge rate (~2-3M data items/s) consistent with value observed at start of run

   Observe active spinners in NewMerger and TapeServer server terminal windows adjacent to web browser

   Chweck MBS terminal data rate is around 20000kB/s Bottom right terminal on desktop 2

7) Desktop 2 - Firefox web browser - TapeServer tab

   Select 'Reload'

   Check data rate consistent with value observed at start of run
  138   Tue Mar 10 10:16:56 2020 OHAIDA Implant Range Scan
A range scan was performed using AIDA.
See the GSI elog: https://elog.gsi.de/despec/S480/19
  137   Tue Mar 10 08:48:07 2020 TD, OH, NH, LS, CA, DK, PJW, RPTuesday 10 March 08.00-16.00
c 09.00 aida03 drops Histogram enable at DAQ GO
        NH determined this was due to Options DB corruption and fixed

09.55 DAQ currently running No Storage mode

ASIC settings 2019Oct31-13.24.23 (different from the December ones yesterday which experienced corruption)
     slow comparator 0xa

BNC PB-5 pulser
     amplitude 1.0V , attenuator x1
     frequency 2Hz
     decay time 1ms

DSSSD biases & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events statistics OK - attachment 3

system wide checks - attachments 4-11
    aida06, 08, 10, 12 re-calibrated OK, all others OK
    aida03 failed WR decoder status - collect all WR errors for baseline - all FEE64s pass WR decoder status
    sync all ASIC clocks using ReSync

merger & TapeServer OK - attachments 12-14

13.29 DAQ to disk file S480/R6

ASIC settings 2019Oct31-13.24.23
     slow comparator 0xa
     aida01 LEC/MEC fast comparator 0x64, all other FEE64s 0xff

14.57 fast comparator thresholds LEC/MEC changed to 0xff for all FEEs

15.01 FEE temperatures normal, aida1 virtex temperature still slightly over 65 degrees (attachment 15)
      leakage currents recorded to spreadsheet (attachment 16)
      good events rate similar to previous, high in DSSD3 and seems to be returning to normal in DSSD1 except in one FEE (attachments 17 and 18)
      system wide checks OK
      data rate around 22000 KB per second (attachment 19)
as of 15.12 on file R6_80

attachment 20 is total event stats shows events are being implanted
attachment 21 and 22 show low energy spectrum left most peak is pulser and rightmost is central noise. middle peak in aida1
attachment 23 and 24 show high energy spectrum, aida4 histogram not being produced unsure why.
attachment 25 and 26 show waveforms, only aida3 histogram working

16.00 checked ASIC control, all OK

16.35 spill is now changed to 10s with beam intensity increase by factor of 10. 3000 implants per spill.

16.42 leakage currents recorded to spreadsheet

17.10 system wide checks ok
      fee64 temperatures ok - attachment 27
      detector bias/leakage currents ok - attachment 28
      good event statistics ok - attachment 29

17.15 moved slit at S2 - rate drop to try remove some 96Pd

17.20 writing to file R6_166

17.44 merger/tapeserver ok. Data rate ~ 25000 kb/s

18:10 Fine tuning implantation profile
      Removing 0.25mm Al degrader (60 mg/cm^2) to get 96Pd slightly further in to AIDA.  
      DESPEC is closing the current file, R6_203 was just finished on AIDA on typing this.
      (Maybe the end  of run S480f004_ on DESPEC side but that elog was not updated since 16:40?)
      Decision based on assumption of 6 GeV 96Pd having a range 1 mm in silicon.

19.16 system wide checks ok
      good event statistics ok - attachment 30
      detector bias/leakage currents ok - attachment 31
      fee64 temperatures ok - attachment 32
      
19.39 TapeServer/Merger ok. Writing to file R6_265. Data rate ~ 25000 kb/s

20.07 1.8.H high energy spectra front and back side (attachments 33 + 34)

20.36 S4 timing change, increased by factor of 2

20.38 Merger/TapeServer ok. Data being forwarded to MBS.

20.38 writing to file R6_308 at rate of ~ 25000 kb/s

21:21 system wide checks ok
      good event statistics ok - attachment 35
      fee64 temperatures ok - attachment 36
      detector bias/leakage currents ok - attachment 37

21:57 TapeServer/merger ok
      data forwarding to MBS ok
      writing to file R6_368 at rate of ~26000 kb/s

23:10 system wide checks ok
      good event statistics ok - attachment 38
      fee64 temperatures ok - attachment 39
      detector bias/leakage currents ok - attachment 40

23:24 TapeServer/merger ok
      data forwarding to MBS ok
      writing to file R6_435 at rate of 27000 kb/s

23:42 rate spectra - attachment 41











 
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 9.png
9.png
Attachment 10: 10.png
10.png
Attachment 11: 11.png
11.png
Attachment 12: 12.png
12.png
Attachment 13: 13.png
13.png
Attachment 14: 14.png
14.png
Attachment 15: 200310_1419_feetemps.png
200310_1419_feetemps.png
Attachment 16: 200310_1451_leakcurrents.png
200310_1451_leakcurrents.png
Attachment 17: 200310_1450_goodevents.png
200310_1450_goodevents.png
Attachment 18: 200309_1506goodevents.png
200309_1506goodevents.png
Attachment 19: 200309_1510_datarate.png
200309_1510_datarate.png
Attachment 20: 200310_1517_stathistograms.png
200310_1517_stathistograms.png
Attachment 21: 200310_1517_lowEoddfees.png
200310_1517_lowEoddfees.png
Attachment 22: 200310_1522_lowEevevfees.png
200310_1522_lowEevevfees.png
Attachment 23: 200310_1523_highEoddfees.png
200310_1523_highEoddfees.png
Attachment 24: 200310_1525_highE_evenfees.png
200310_1525_highE_evenfees.png
Attachment 25: 200310_1529_waveformsoddfees.png
200310_1529_waveformsoddfees.png
Attachment 26: 200310_1530_waveformsevenfees.png
200310_1530_waveformsevenfees.png
Attachment 27: 200310_1707_temp.png
200310_1707_temp.png
Attachment 28: 200310_1708_bias.png
200310_1708_bias.png
Attachment 29: 200320_1708_stats.png
200320_1708_stats.png
Attachment 30: 200310_1915_stats.png
200310_1915_stats.png
Attachment 31: 200310_1916_bias.png
200310_1916_bias.png
Attachment 32: 200310_1919_temp.png
200310_1919_temp.png
Attachment 33: 200310_2000_18h_odd.png
200310_2000_18h_odd.png
Attachment 34: 200310_2005_18h_even.png
200310_2005_18h_even.png
Attachment 35: 200310_2119_stats.png
200310_2119_stats.png
Attachment 36: 200310_2119_temp.png
200310_2119_temp.png
Attachment 37: 200310_2120_bias.png
200310_2120_bias.png
Attachment 38: 200310_2307_temp.png
200310_2307_temp.png
Attachment 39: 200310_2308_stats.png
200310_2308_stats.png
Attachment 40: 200310_2309_bias.png
200310_2309_bias.png
Attachment 41: 200310_2342_rates.png
200310_2342_rates.png
  136   Mon Mar 9 13:50:08 2020 LS, CA, DK09/03/2020 system checks
14:52 Attachment 1 shows low energy spectra. AIDA06 shows strange baseline, possible double-hit?

15:06 Systems checks fine.
      Master clock failed (no master clock)
      all checks passed
      memory checks all around 38k as normal

      Attachment 2: Temperature check all normal except aida01 virtex temp slightly over 65 degrees

      Attachment 3: Good event stats seem normal except downstream SSD seems to be high around 200k

      Attachment 4: leakage currents seem normal at expected values

15:16 Merger working for all FEEs
      tape server is on but no storage
      data rate is around 25 MB per second (high but consistent with the high stats of downstream SSD)

ASIC controls checked and are all OK

15.48 low energy pulsar peak FWHM

      fee   FHWM
      1     89.08
      2     132.06
      3     70.73
      4     81.84
      5     62.86
      6     79.92
      7     122.13
      8     73.31
      9     198.39
      10    115.53
      11    100.55
      12    144.04

ALL FEEs in downstream SSD are high

16:21 included low energy pulsar peaks (Attachments 5 & 6) and waveforms for all FEEs (Attachments 7 & 8)

16:25 included rates, note not to common scale in y (attachment 9)

ASIC settings file 2019Dec19-16.19.51 but some others are on 2019Oct31-13.24.23
  Did not seem completely reproducible?
  We carefully checked all the individual FEE and ASIC settings, and they are all the same (except shaping reference), with NO 0xad

Now we made sure they are all using 2019Dec19-16.19.51 (which we had saved first on a good one)


19:15 Update

There is a safety interlock box in S4 to monitor humidity, dew point, temperature, etc for safety.  
It had a loose solder connection, and when someone moved it out of the way, the interlock was tripped.
After some debugging, the interlock is now in a more stable condition, but a lot of power was cut from AIDA systems

One unusual thing was that one of the RPi systems got a full /var/messages (or similar) and ate all the available disk space 
  This meant that ssh -X (to, e.g., activate putty) could not work, giving a permission denial error (ssh connection was allowed, but not X11 forwarding)

Now we have brought back up the AIDA systems and should run through the checks

Most system checks look good (except some calibration errors)

Temps were running high until we reloaded the ASIC settings, then the temperatures began to cool

Bias and leak currents attached as #10

Temps as #11

Stats as #12

19:50 Started writing to disk so that we can do an implantation depth profile. Will analyse near online using AIDASort.

21:45 Stopped writing to file -> Runs where AIDA thickness were investigated corresponds to R3_62 to R3_72
      A rough method using the rates histogram was used to judge roughly when depositing in each histogram - see attachment 12 (Stopping in 2)
      Can match to the FRS files to work out degrader thickness with timestamps (File stopped at Mon 09 Mar 2020 09:44:51 PM CET

21:51 AIDA running to no storage again
Attachment 1: 200309_lowE_spectra.png
200309_lowE_spectra.png
Attachment 2: 200309_FEEtemp.png
200309_FEEtemp.png
Attachment 3: 200309_stats.png
200309_stats.png
Attachment 4: 200309_leakcurrents.png
200309_leakcurrents.png
Attachment 5: 200309_lowE_pulsarpeaksodd.png
200309_lowE_pulsarpeaksodd.png
Attachment 6: 200309_lowE_pulsarpeakseven.png
200309_lowE_pulsarpeakseven.png
Attachment 7: 200309_waveformsodd.png
200309_waveformsodd.png
Attachment 8: 200309_waveformseven.png
200309_waveformseven.png
Attachment 9: 200309_rates.png
200309_rates.png
Attachment 10: biases_after_trip.png
biases_after_trip.png
Attachment 11: temps-after-crash.png
temps-after-crash.png
Attachment 12: stats-after-crash.png
stats-after-crash.png
Attachment 13: ImplantRangeRough.png
ImplantRangeRough.png
  135   Mon Mar 9 12:47:18 2020 NHReport: Unusual AIDA Timestamp Error
Noted using my AIDA event builder code, a timewarp was reported and I noticed the ADC event had a different timestamp LSB to the "WR" markers preceding it, which I didn't think was possible in the merger as it stands

 aida event 875015fa 0f68098c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68098c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68098c
 aida event c1f6851b 0f68098c | ADC 08:54 L 851b
 event time: 15fa4b164f68098c
 aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68115c
 aida event c1ef8238 0f68115c | ADC 08:47 L 8238
 event time: 15fa4b164f68115c
 aida event 875015fa 0f68115c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68115c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68115c
 aida event c1f884b0 0f68115c | ADC 08:56 L 84b0
 event time: 15fa4b164f68115c
 aida event 8a5015fa 0f681382 | INFO 5 SYNC6348 11 015fa
 aida event 8a44b164 0f681382 | INFO 4 SYNC4828 11 4b164
 event time: 15fa4b164f681382
 aida event c2817dc8 0f681382 | ADC 11:01 L 7dc8
 event time: 15fa4b164f681382
 aida event 875015fa 0f68192c | INFO 5 SYNC6348 08 015fa
 aida event 8744b164 0f68192c | INFO 4 SYNC4828 08 4b164
 event time: 15fa4b164f68192c
 aida event c1f9849c 0f68192c | ADC 08:57 L 849c
 event time: 15fa4b164f68192c
 aida event 815015fa 0f681f94 | INFO 5 SYNC6348 02 015fa
 aida event 8144b164 0f681f94 | INFO 4 SYNC4828 02 4b164
 event time: 15fa4b164f681f94
 aida event c0568084 04b6854a | ADC 02:22 L 8084
 event time: 15fa4b1644b6854a
AIDA Timewarp (15fa4b1644b6854a before 15fa4b164f68192c)
Event reported errors, skipping this file...
Events: 506459240   506459240             (0 errors)

Unsure if the cause, easily handled by ignoring such events but should be "impossible"
  134   Sat Feb 29 10:56:38 2020 NHAlpha Hit Pattern
Circa 700 GB of data

A few channels in DSSD 1 missing? Otherwise seems OK
Attachment 1: alphas_1d.pdf
alphas_1d.pdf
  133   Tue Feb 25 09:55:59 2020 NHHV Scope Traces
Looking at the HV bias in a scope to see if it is the source of noise especially on the p+n side

Channel 1: HV Core channel 4
Channel 2: HV Braid Channel 4
Math: Ch1 - Ch2

100V bias (100 uA leakage current for 1 MOhm)

No obvious frequency components or large frequency components?
I do not know how this compares to RIKEN or STFC-DL

Updated to put the right frequency range (misread before)

Fig 3-4: Normalish looking data 

Fig 5-6: Examples of noise appearing randomly on math line

Fig 7: 1ms long trace, noise stopped appearing

Update 26.02.2020

Fig 8: Trace with all 4 HV channels off

Fig 9: Trace with only oscilloscope biased

Fig 10: Trace with braids unplugged from all 3 FEE64s (only channel 4 connected)

Things to try:

Joining clean earth & chassis ground?

Checking HV jumper?

Finding 'scope probes and testing NIM socket?
Attachment 1: SCRN0057.PNG
SCRN0057.PNG
Attachment 2: SCRN0054.PNG
SCRN0054.PNG
Attachment 3: SCRN0059.PNG
SCRN0059.PNG
Attachment 4: SCRN0060.PNG
SCRN0060.PNG
Attachment 5: SCRN0071.PNG
SCRN0071.PNG
Attachment 6: SCRN0073.PNG
SCRN0073.PNG
Attachment 7: SCRN0075.PNG
SCRN0075.PNG
Attachment 8: SCRN0078.PNG
SCRN0078.PNG
Attachment 9: SCRN0079.PNG
SCRN0079.PNG
Attachment 10: SCRN0080.PNG
SCRN0080.PNG
  131   Fri Feb 21 12:16:06 2020 TD, NHReport - high - unusual FEE64 crash during startup
Attachment 1: ttyUSB4
06/15:20:21|
ISOL Version 1.00 Date 9th January 2017
06/15:20:21|
Flash base address=FC000000
06/15:20:21|
Set Flash to ASync Mode
06/15:20:21|
XST_SUCCESS
06/15:20:21|
Finished copying zImage to RAM 
06/15:20:21|
06/15:20:21|
Found 0 errors checking kernel image
06/15:20:22|VHDL version number 0X12110103 
06/15:20:22|
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
06/15:20:22|
Starting LMK 3200 setup 
06/15:20:22|
06/15:20:22|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
06/15:20:22|
.... SPI Base Address=0x81400000
06/15:20:22|
clk_control_reg=0x4
06/15:20:22|Next step is SPIconfig
06/15:20:23|
Control 32(0x81400000)=0x180
06/15:20:23|
SlaveSel(0x81400000)=0x3
06/15:20:23|
Ctrl(0x81400000)=0xE6
06/15:20:23|
Ctrl(0x81400000)=0x86
06/15:20:23|SPIconfig done now to set up the LMK3200 registers
06/15:20:23|LMK #0 : regInit[0]=0x80000000
06/15:20:23|LMK #0 : regInit[1]=0x10070600
06/15:20:23|LMK #0 : regInit[2]=0x60601
06/15:20:23|LMK #0 : regInit[3]=0x60602
06/15:20:23|LMK #0 : regInit[4]=0x60603
06/15:20:23|LMK #0 : regInit[5]=0x70624
06/15:20:23|LMK #0 : regInit[6]=0x70605
06/15:20:23|LMK #0 : regInit[7]=0x70606
06/15:20:23|LMK #0 : regInit[8]=0x70627
06/15:20:23|LMK #0 : regInit[9]=0x10000908
06/15:20:23|LMK #0 : regInit[10]=0xA0022A09
06/15:20:23|LMK #0 : regInit[11]=0x82800B
06/15:20:23|LMK #0 : regInit[12]=0x28C800D
06/15:20:23|LMK #0 : regInit[13]=0x830020E
06/15:20:23|LMK #0 : regInit[14]=0xC800180F
06/15:20:23|
Calibrate completed at 942 counts
06/15:20:23|
Setting Clock Control =0x0000000B, to set GOE and sync bit
06/15:20:23|
Ctrl @ SPIstop (0x81400000)=0x186
06/15:20:23|
0 delay setup completed at 0 counts
06/15:20:23|
Finished Clock setup LMK03200
06/15:20:23|
completed LMK 3200 setup 
06/15:20:23|
Loaded all four ASICs with default settings 
06/15:20:23|
Setting the ADCs into calibration mode 
06/15:20:23|
06/15:20:23|
Control 32(0x81400400)=0x180
06/15:20:24|
SlaveSel(0x81400400)=0xFF
06/15:20:24|
Ctrl(0x81400400)=0xE6
06/15:20:24|
Ctrl(0x81400400)=0x86
06/15:20:24|
Init : Config of AD9252 SPI ok
06/15:20:24|
06/15:20:24|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
06/15:20:24|
ADCs calibrated
06/15:20:24|
06/15:20:24|
Control 32(0x81400400)=0x186
06/15:20:24|
SlaveSel(0x81400400)=0xFF
06/15:20:24|
Ctrl(0x81400400)=0xE6
06/15:20:24|
Ctrl(0x81400400)=0x86Config of AD9252 SPI ok
06/15:20:24|
06/15:20:24|
Ctrl @ SPIstop (0x81400400)=0x186Jumping to kernel simpleboot...
06/15:20:24|
06/15:20:24|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
06/15:20:24|
Allocating 0x3b78cc bytes for kernel ...
06/15:20:24|
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
06/15:20:27|
06/15:20:27|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
06/15:20:34|
Finalizing device tree... flat tree at 0xbd1300
06/15:20:34|
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xd8 0x21 
06/15:20:34|Using Xilinx Virtex440 machine description
06/15:20:34|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
06/15:20:34|Zone PFN ranges:
06/15:20:34|  DMA      0x00000000 -> 0x00007000
06/15:20:34|  Normal   0x00007000 -> 0x00007000
06/15:20:34|Movable zone start PFN for each node
06/15:20:34|early_node_map[1] active PFN ranges
06/15:20:34|    0: 0x00000000 -> 0x00007000
06/15:20:34|MMU: Allocated 1088 bytes of context maps for 255 contexts
06/15:20:35|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
06/15:20:35|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
06/15:20:35|PID hash table entries: 512 (order: 9, 2048 bytes)
06/15:20:35|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
06/15:20:35|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
06/15:20:35|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
06/15:20:35|Kernel virtual memory layout:
06/15:20:35|  * 0xffffe000..0xfffff000  : fixmap
06/15:20:35|  * 0xfde00000..0xfe000000  : consistent mem
06/15:20:35|  * 0xfde00000..0xfde00000  : early ioremap
06/15:20:35|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
06/15:20:35|NR_IRQS:512
06/15:20:35|clocksource: timebase mult[a00000] shift[22] registered
06/15:20:35|Console: colour dummy device 80x25
06/15:20:35|Mount-cache hash table entries: 512
06/15:20:35|NET: Registered protocol family 16
06/15:20:35|PCI: Probing PCI hardware
06/15:20:35|bio: create slab <bio-0> at 0
06/15:20:35|NET: Registered protocol family 2
06/15:20:35|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
06/15:20:36|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
06/15:20:36|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
06/15:20:36|TCP: Hash tables configured (established 4096 bind 4096)
06/15:20:36|TCP reno registered
06/15:20:36|NET: Registered protocol family 1
06/15:20:36|ROMFS MTD (C) 2007 Red Hat, Inc.
06/15:20:36|msgmni has been set to 214
06/15:20:36|io scheduler noop registered
06/15:20:36|io scheduler anticipatory registered
06/15:20:36|io scheduler deadline registered
06/15:20:36|io scheduler cfq registered (default)
06/15:20:36|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
06/15:20:36|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
06/15:20:36|console [ttyS0] enabled
06/15:20:36|brd: module loaded
06/15:20:36|loop: module loaded
06/15:20:36|Device Tree Probing 'ethernet'
06/15:20:36|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:d8:21
06/15:20:36|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
06/15:20:36|XLlTemac: DCR address: 0x80
06/15:20:36|XLlTemac: buffer descriptor size: 32768 (0x8000)
06/15:20:36|XLlTemac: Allocating DMA descriptors with kmalloc
06/15:20:37|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
06/15:20:37|XTemac: PHY detected at address 7.
06/15:20:37|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
06/15:20:37|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37| Intel/Sharp Extended Query Table at 0x010A
06/15:20:37|Using buffer write method
06/15:20:37|cfi_cmdset_0001: Erase suspend on write enabled
06/15:20:37|cmdlinepart partition parsing not available
06/15:20:37|RedBoot partition parsing not available
06/15:20:37|Creating 5 MTD partitions on "fc000000.flash":
06/15:20:37|0x000000000000-0x000000500000 : "golden_firmware"
06/15:20:37|0x000000500000-0x000000800000 : "golden_kernel"
06/15:20:37|0x000000800000-0x000000d00000 : "user_firmware"
06/15:20:37|0x000000d00000-0x000000fe0000 : "user_kernel"
06/15:20:37|0x000000fe0000-0x000001000000 : "env_variables"
06/15:20:38|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
06/15:20:38|SPI: XIlinx spi: bus number now 32766
06/15:20:38|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
06/15:20:38|SPI: XIlinx spi: bus number now 32765
06/15:20:38|mice: PS/2 mouse device common for all mice
06/15:20:38|Device Tree Probing 'i2c'
06/15:20:38| #0 at 0x81600000 mapped to 0xD1030000, irq=22
06/15:20:38|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
06/15:20:38|TCP cubic registered
06/15:20:38|NET: Registered protocol family 17
06/15:20:38|RPC: Registered udp transport module.
06/15:20:38|RPC: Registered tcp transport module.
06/15:20:38|eth0: XLlTemac: Options: 0x3fa
06/15:20:39|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
06/15:20:39|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
06/15:20:39|eth0: XLlTemac: speed set to 1000Mb/s
06/15:20:41|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
06/15:20:41|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
06/15:20:41|Sending DHCP requests ., OK
06/15:20:42|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.3
06/15:20:42|IP-Config: Complete:
06/15:20:42|     device=eth0, addr=192.168.11.3, mask=255.255.255.0, gw=255.255.255.255,
06/15:20:42|     host=aida03, domain=dl.ac.uk, nis-domain=nuclear.physics,
06/15:20:42|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida03
06/15:20:42|Looking up port of RPC 100003/2 on 192.168.11.99
06/15:20:42|Looking up port of RPC 100005/1 on 192.168.11.99
06/15:20:42|VFS: Mounted root (nfs filesystem) on device 0:12.
06/15:20:43|Freeing unused kernel memory: 168k init
06/15:20:43|
INIT: version 2.86 booting

06/15:20:43|Starting sysinit...
06/15:20:44|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
06/15:20:45|		Press 'I' to enter interactive startup.
06/15:20:45|Setting clock  (utc): Thu Sep  6 14:17:26 BST 2018 [  OK  ]

06/15:20:46|Building the cache [  OK  ]

06/15:20:46|Setting hostname aida03:  [  OK  ]

06/15:20:48|Mounting local filesystems:  [  OK  ]

06/15:20:49|Enabling /etc/fstab swaps:  [  OK  ]
06/15:20:52|Finishing sysinit...
06/15:20:53|
INIT: Entering runlevel: 3

06/15:20:55|Entering non-interactive startup
06/15:20:56|FATAL: Module ipv6 not found.
06/15:20:57|Bringing up loopback interface:  [  OK  ]

06/15:20:59|FATAL: Module ipv6 not found.
06/15:20:59|Starting system logger: [  OK  ]

06/15:21:00|Starting kernel logger: [  OK  ]

06/15:21:00|Starting rpcbind: [  OK  ]

06/15:21:08|Mounting NFS filesystems:  [  OK  ]

06/15:21:12|Mounting other filesystems:  [  OK  ]

06/15:21:15|Starting xinetd: [  OK  ]

06/15:21:17|Starting midas:  Starting MIDAS Data Acquisition for aida03
06/15:21:18|xaida: device parameters: base=0x81000000 size=0x200000
06/15:21:24|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
06/15:21:24|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
06/15:21:24|xaida: driver assigned major number 254
06/15:21:24|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
06/15:21:31|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
06/15:21:31|AIDAMEM: aidamem: driver assigned major number 253
06/15:21:31|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
06/15:21:38|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
06/15:21:38|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
06/15:21:38|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
06/15:21:38|Computer Name = aida03; Temp Directory = /tmp/tcl361
06/15:21:42|package limit is not available: can't find package limit
06/15:21:45|Running with default file descriptor limit
06/15:21:45|package setuid is not available: can't find package setuid
06/15:21:46|Running as user 0 group 0
06/15:21:46|
06/15:21:47|AIDA Data Acquisition Program Release 8_57.260418 starting
06/15:21:47|
06/15:21:47|Built without pthreads
06/15:21:47|
06/15:21:47|xaida: open:
06/15:21:48|CreatinAIDAMEM: aidamem_open:
06/15:21:48|g NetVars
06/15:21:48|Output buffer length = 65504; format option = 4; transfer option = 3
06/15:21:48|EB transfer option = 3
06/15:21:48|NetVars created and initialised
06/15:21:48|Statistics thread starting
06/15:21:48|Statistics thread created
06/15:21:48|Stat/Rate creation thread starting
06/15:21:48|Data Acquisition task has PID 375
06/15:21:48|Stat/Rate creation thread created
06/15:21:48|Hit/Rate creation thread starting
06/15:21:48|Hit/Rate creation thread created
06/15:21:48|AIDA Heartbeat thread starting
06/15:21:48|Heartbeat thread created
06/15:21:48|Installing signal handlers
06/15:21:48|Done
06/15:21:48|ModuleNum = 0
06/15:21:48|Aida Initialise complete. AidaExecV8_57: Build 260418. HDL version : 12110103 
06/15:21:48|Spectra table initialised
06/15:21:48|AIDA Data Acquisition now all ready to start
06/15:21:48|SIGBUS, SIGSEGV and SIGPIPE traps setup
06/15:21:48|/debug user "debug" password "qy oblg2zuiw"
06/15:21:55|httpd started on port 8015
06/15:21:55|
06/15:21:55|Cannot use /MIDAS/config/TclHttpd/aida03@8015/startup.tcl
06/15:21:56|Custom startup from /MIDAS/config/TclHttpd/aida03/startup.tcl
06/15:21:56|XAIDA Access package 1.0
06/15:21:57|/XAIDAAccessServer
06/15:21:57|XAD9252 Access package 1.0
06/15:21:57|/XAD9252AccessServer
06/15:21:58|/DataBaseAccessServer
06/15:21:58|[  OK  ]

06/15:21:58|/NetVarService
06/15:21:58|/SigTaskService
06/15:21:58|Loaded MemSasAccess
06/15:21:58|/SpectrumService
06/15:21:58|loading tcl/AIDARunControl.tcl for namespace ::
06/15:21:59|/DataAcquisitionControlServer
06/15:21:59|
06/15:21:59|DefineMessage unknown
06/15:21:59|DENX ELDK version 4.2 build 2008-04-01
06/15:21:59|Linux 2.6.31 on a ppc
06/15:21:59|
06/15:21:59|aida03 login: Run Control Server Implementation for AIDA
06/15:21:59|RunControlServer loaded
06/15:21:59|loading Html/RunControl/implementation.tcl
06/15:21:59|RunControl loaded
06/15:22:00|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
06/15:22:00|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
06/15:22:00|created SigTask registers 0 0  0 0
06/15:22:00|Created UI registers
06/15:22:00|EXPERIMENTS/AIDA/Options/aida03 Stat.offset NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:01|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_W_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:01|EXPERIMENTS/AIDA/Options/aida03 TS_SYNC_PHASE NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:02|EXPERIMENTS/AIDA/Options/aida03 ExtClk NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:02|EXPERIMENTS/AIDA/Options/aida03 Aida.shift NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:02|EXPERIMENTS/AIDA/Options/aida03 MACB_TRIG_MODE NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:02|EXPERIMENTS/AIDA/Options/aida03 Aida.offset NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:03|EXPERIMENTS/AIDA/Options/aida03 Aida_GroupBase NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:03|EXPERIMENTS/AIDA/Options/aida03 Rate.channels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:03|EXPERIMENTS/AIDA/Options/aida03 Stat.channels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:03|EXPERIMENTS/AIDA/Options/aida03 WAVE_DMA_HWM NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:04|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_D_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:04|EXPERIMENTS/AIDA/Options/aida03 ASIC.settings NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:04|EXPERIMENTS/AIDA/Options/aida03 Aida.Vchannels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:04|EXPERIMENTS/AIDA/Options/aida03 Aida.Wchannels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:05|EXPERIMENTS/AIDA/Options/aida03 Stat.shift NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:05|EXPERIMENTS/AIDA/Options/aida03 Aida.channels NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:05|EXPERIMENTS/AIDA/Options/aida03 Include.Aida NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:05|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_V_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:06|EXPERIMENTS/AIDA/Options/aida03 DataFormat NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:06|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_H_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:06|EXPERIMENTS/AIDA/Options/aida03 DataAcqPgm NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:06|EXPERIMENTS/AIDA/Options/aida03 ASIC_DMA_HWM NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:07|EXPERIMENTS/AIDA/Options/aida03 Aida_Hist_L_Enable NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:07|EXPERIMENTS/AIDA/Options/aida03 Index NOT available: error 0x10004; unable to access CONTENTS for node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03: couldn't open "/MIDAS/DB/EXPERIMENTS/AIDA/Options/aida03/CONTENTS": permission denied
06/15:22:07|AIDA RunControl loaded
06/15:22:07|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
06/15:22:07|
06/16:01:02|
ISOL Version 1.00 Date 9th January 2017
06/16:01:02|
Flash base address=FC000000
06/16:01:02|
Set Flash to ASync Mode
06/16:01:02|
XST_SUCCESS
06/16:01:02|
Finished copying zImage to RAM 
06/16:01:02|
06/16:01:02|
Found 0 errors checking kernel image
06/16:01:03|VHDL version number 0X12110103 
06/16:01:03|
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
06/16:01:04|
Starting LMK 3200 setup 
06/16:01:04|
... 127179 more lines ...
  130   Fri Feb 21 09:39:38 2020 TD, NHFriday 21 February
Attachment 1 - layout 2 - stat spectra
Attachment 2 - detector biases & leakage currents - OK
Attachment 3 - FEE64 temperatures - OK
Attachments 4-7 - TapeServer, Merger, Merger Statistics, good event statistics - OK

10:40 ASIC check load x2


13.12 DAQ stopped c. 10.50
      unable to connect messages for multiple FEE64s

      After lunch powwer-cycled FEE64s and restarted DAQ
      aida03 crashed (see https://elog.ph.ed.ac.uk/DESPEC/131 ) and rebooted by issuing reboot command as root

      Data file 20FEB20/R1
      ASIC settings 2019Oct31-13.24.23
      slow comparator 0xa -> 0x64
      BNC PB-5 pulser OFF
      continue alpha background run
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
  129   Fri Feb 21 09:21:10 2020 NH(Early) Alpha Analysis
Overnight alpha data

======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Thu Feb 20 17:46:47 2020
Last event : Fri Feb 21 10:07:57 2020
Duration   : 58869.9 seconds
--------------------------------------
MBS Events   (M):     1248648 (21 Hz)
AIDA Words   (W): 10222680133 (27 kHz)
ADC Words    (A):  3407431088 (-15076 Hz)
-- Decays    (D):  3407017078 (-15083 Hz)
-- Implants  (I):      414010 (7 Hz)
Info Words   (F):  6815249045 (-30146 Hz)
-- SYNC48    (S):  3407559635 (-15074 Hz)
-- SYNC63    (s):  3407559635 (-15074 Hz)
-- PAUSE     (P):         614 (0 Hz)
-- RESUME    (R):         614 (0 Hz)
-- Discrim   (D):      128547 (2 Hz)
-- Unknown   (U):           0 (0 Hz)
Timewarps    (T):          79 (0 Hz)
Dead Time    (X): 26.8658 s (0.04563588%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |           W           A           D           I           F           S           s     P     R       D     U     T  X
   1 |        5025        1246        1093         153        3779        1675        1675     0     0     429     0     0  0
   2 |        6180         991         851         140        5189        2060        2060     0     0    1069     0     0  0
   3 |  7692461404  2564028610  2564028286         324  5128432794  2564153392  2564153392   614   614 	124782     0    58 26.8658
   4 |        2277         642         581          61        1635         759         759     0     0     117     0     0  0
   5 |        6174        1249        1010         239        4925        2058        2058     0     0     809     0     0  0
   6 |     2467062      822244      822191          53     1644818      822354      822354     0     0     110     0     0  0
   7 |        2484         740         701          39        1744         828         828     0     0      88     0     0  0
   8 |   358445004   119481621   119069026      412595   238963383   119481668   119481668     0     0      47     0     1  0
   9 |        7134        1530        1243         287        5604        2378        2378     0     0     848     0     0  0
  10 |  2168031885   722677295   722677295           0  1445354590   722677295   722677295     0     0       0     0    20  0
  11 |        3981        1252        1190          62        2729        1327        1327     0     0      75     0     0  0
  12 |     1241523      413668      413611          57      827855      413841      413841     0     0     173     0     0  0
**** DESPECAnalysis: PostLoop

A few FEEs running "hot", maybe check. 3 had some alignment issues may need rechecking.
Timewarps somehow in aida03 and aida10 but may be code mistake - must check.
Only aida03 ran fast enough to have dead time.

aida10 shows decay data now - DSSD seems ok

Fig 1: DSSD 1D hit pattern 

Fig 2: DSSD Energy

Fig 3: Single channel in aida03 that is hot

Fig 4: Channels 48-64 in aida08 are hot (wiring?)

Fig 5: Channels 17, 19 (crosstalk?) and 27, 33, 38, 39

All otehr channels quiet and OK. 
Attachment 1: DSSD_1d_HP.png
DSSD_1d_HP.png
Attachment 2: DSSD_E.png
DSSD_E.png
Attachment 3: FEE3_43_Hot.png
FEE3_43_Hot.png
Attachment 4: fee8_hot.png
fee8_hot.png
Attachment 5: Fee10_hot.png
Fee10_hot.png
  128   Thu Feb 20 16:35:55 2020 TD, NHThu 20 February 2020
DSSD#3 (most downstream) was replaced

Noted that old DSSD wafer was not sitting properly - detached - in PCB, PCB flexing and/or inadequate epoxy gluing

New DSSD#3 - 2998-22 - 1006 um - 100 V - ~4 uA

Old Detector removed was 3208-14

Fig 1. HV Control

Fig 2. Temps

Fig 3. Stats

Fig 4. Rates

Fig 5-6: Waves

Fig 7-8: Pulser Peaks

Start alpha run at 17:48, 20FEB20, R1
 Slow comparator 0x64
 Pulser off

Fig 9 : Alpha stats

Fig 10: Merger stats

Fig 11: Merger main page

Fig 12: Timestamp errors in merger console
Attachment 1: hv.png
hv.png
Attachment 2: temps.png
temps.png
Attachment 3: stats.png
stats.png
Attachment 4: rates.png
rates.png
Attachment 5: waves_even.png
waves_even.png
Attachment 6: waves_odd.png
waves_odd.png
Attachment 7: pulser_even.png
pulser_even.png
Attachment 8: pulser_odd.png
pulser_odd.png
Attachment 9: alpha_stats.png
alpha_stats.png
Attachment 10: merger.png
merger.png
Attachment 11: merger_stats.png
merger_stats.png
Attachment 12: merger_errors.png
merger_errors.png
  127   Wed Feb 19 10:58:29 2020 NH, PJCSReport: aida09 Kernel Panic & Lost WR
> aida09 crashed over the weekend and automatically rebooted.
> After the reboot the WR timestamp sent to the merger is in the future and hence incorrect
> 
> Reset/Setup did not fix issue
> Sync ASICs did not fix issue
> GSI White Rabbit control page shows a correct WR timestamp
> 
> Attach 1: ttyUSB12 (aida09 log with kernel panic)
> Attach 2: GSI White Rabbit control page
> Attach 3: "Collect All WR Timestamps"
> Attach 4: RAW Display for aida09
>  
> WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000
> 
> WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
> c.f. "GSI page" timestamp starting 0x15F427F5 
> 
> Attach 5: Timestamp shown by merger

Tested aida01 and aida09 today ( 19/2/2020 ) and both make sense relative to their Timestamps.
It is the case that the WR timestamp from the "GSI White Rabbit Timestamp" browser window is direct from the White Rabbit decoder and as such has an LSB of 
1nS and is captured at T0 time ( 10uS intervals ) whereas the timestamp of the SYNC in the raw data display is 10nS LSB and is captured at the time of a 
logical "rollover" of the lower 14 bits of this 10nS timestamp.


Is it the case that the system has been power cycled since aida09 got its timestamp wrong ?


It is possible to reset an individual FEE64 WR decoder by writing 0x80 into register 0 of the individual "GSI White Rabbit Timestamp" page. Then 0x1 to re-
enable the decoder.
This should never be necessary as the decoder should be collecting the latest timestamp continuosly.

The statement remains true however that if the Linux in a FEE64 has a "Panic" then the FEE64 must be powercycled in order for the subsequent data to be 
considered reliable.
The Raspberry Pi Console control browser will count the number of "Panics" in the console logged text files so they can be monitored.

If this occurs again then the system wide check results would be interesting.
  126   Mon Feb 17 09:48:34 2020 NHReport: aida09 Kernel Panic & Lost WR
aida09 crashed over the weekend and automatically rebooted.
After the reboot the WR timestamp sent to the merger is in the future and hence incorrect

Reset/Setup did not fix issue
Sync ASICs did not fix issue
GSI White Rabbit control page shows a correct WR timestamp

Attach 1: ttyUSB12 (aida09 log with kernel panic)
Attach 2: GSI White Rabbit control page
Attach 3: "Collect All WR Timestamps"
Attach 4: RAW Display for aida09
 
WR Time Item 0x80500232 0x0de48000; Time (48:63)=0x232; Time (28:47)=0x20310; Time (0:27)=0x0de48000
WR Time Item 0x80420310 0x0de48000; Time (28:47)=0x20310; Time (0:27)=0x0de48000

WR Timestamp = 0x23220310de48000 * 10 = 0x15F541EA 8AED0000 = 2020-02-21 CET 01:01:59.699537920
c.f. "GSI page" timestamp starting 0x15F427F5 

Attach 5: Timestamp shown by merger
Attachment 1: ttyUSB12
Attachment 2: GSI_WR.png
GSI_WR.png
Attachment 3: GSI_WR2.png
GSI_WR2.png
Attachment 4: GSI_WR4.png
GSI_WR4.png
Attachment 5: GSI_WR3.png
GSI_WR3.png
  125   Fri Feb 14 12:50:29 2020 NHAIDA Noise 14.02.2020
Electricians changed the grounding for the DESPEC platform to a large pillar that goes deep into soil

Change in grounding was apparently successful at reducing the noise complaints in Cave C (R3B)

AIDA:

Fig 1 & 2: Waveforms (Odd/Even)
Fig 3 & 4: Waveforms Zoomed
Fig 5: Good Event Rate
Fig 6: Merger Rate: 2 M items/sec
Fig 7: HV supply (no change)

There does seem to be a modest improvement (Merger rate down from 3 million, good event rates seem *better*)
(Reminder: AIDA@GSI sends 3 items per each ADC event, so merger rate is higher)
Not sure if this is still where it should stand?
Attachment 1: wavesOdd.png
wavesOdd.png
Attachment 2: wavesEven.png
wavesEven.png
Attachment 3: wavesOddZ.png
wavesOddZ.png
Attachment 4: stats_goodevents.png
stats_goodevents.png
Attachment 5: mergerstats.png
mergerstats.png
Attachment 6: hv.png
hv.png
  124   Wed Jan 29 19:09:51 2020 NHaida10 troubleshooting
> aida10 has no ASIC events when running without pulser
> 
> DSSD ribbon cable was replaced, no change
> 
> 29.01.2020 - change AIDA half adapter board, no change
> 
> Swapped HV (braid) from aida10 to aida12 - no change in leakage current, now testing data...
>  Looks the same (no data) fig 2
> 
> Very hot channel in aida08 as well:
>  - Channel 104 in histogram Stat = 4.7.L  fig 1
> 
> Change FEE64 card?
> Any other checks possible?

when running the pulser in 10 do all the channels respond correctly ?
what happens if you drop the shaper threshold into the noise ?
  122   Wed Jan 29 12:22:13 2020 NHaida10 troubleshooting
aida10 has no ASIC events when running without pulser

DSSD ribbon cable was replaced, no change

29.01.2020 - change AIDA half adapter board, no change

Swapped HV (braid) from aida10 to aida12 - no change in leakage current, now testing data...
 Looks the same (no data) fig 2

Very hot channel in aida08 as well:
 - Channel 104 in histogram Stat = 4.7.L  fig 1

Change FEE64 card?
Any other checks possible?

31.01.2020:

Installed new FEE64 card aida10: d8:80:39:41:ba:89
Updated dhcpd.conf
System boots OK.
returned bias ground to aida10 for symmetry reasons
Firmware updated from 0x12110103 to 0x18430701

Left recording alpha data over the weekend:
See stats in fig 3

DSSD 1 and 2 look excellent (one broken strip in DSSd 1)?

aida10 still no counts?
aida09 looks very... weird as well?
aida11 and aida12 seem ok?

Detector fault?
Attachment 1: aida08-noise.png
aida08-noise.png
Attachment 2: aida10-noitems.png
aida10-noitems.png
Attachment 3: Stats.png
Stats.png
  121   Mon Jan 20 14:01:23 2020 NHAIDA Noise 20.01.2019
> Checking noise
> 
> DESPEC crate entirely offline except:
> VME Crate (for VETAR2)
> AIDA (All)
> 
> Figs 1-2: Waveforms (Odd/Even)
> Figs 3-4: Zoomed In (Odd/Even)
> Fig 5: Good Event Stats
> Fig 6: Merger Rate
> 
> Noise still present, despite nothing on DESPEC.
> 
> Things to check: 
> Turn off VME (kills time stamps but might be OK for waveform test?)
> Check floor PSU under AIDA is offline too 
> 
> Not sure that other FRS systems (e.g. Ion Catcher) is offline - I think RF may be on.

Update with different sampling rates (in order)
10 (200 ns)
100 (2 us)
1000 (20 us)

Note that some waveforms didn't update very reliably
In particular aida11 does not seem right at large sample rates
Attachment 1: Odd_10_Z.png
Odd_10_Z.png
Attachment 2: Even_10_Z.png
Even_10_Z.png
Attachment 3: Odd_100_Z.png
Odd_100_Z.png
Attachment 4: Even_100_Z.png
Even_100_Z.png
Attachment 5: Odd_1000_Z.png
Odd_1000_Z.png
Attachment 6: Even_1000_Z.png
Even_1000_Z.png
  120   Wed Jan 15 14:53:41 2020 NHAIDA Noise 14.01.2019
Checking noise

DESPEC crate entirely offline except:
VME Crate (for VETAR2)
AIDA (All)

Figs 1-2: Waveforms (Odd/Even)
Figs 3-4: Zoomed In (Odd/Even)
Fig 5: Good Event Stats
Fig 6: Merger Rate

Noise still present, despite nothing on DESPEC.

Things to check: 
Turn off VME (kills time stamps but might be OK for waveform test?)
Check floor PSU under AIDA is offline too 

Not sure that other FRS systems (e.g. Ion Catcher) is offline - I think RF may be on.
Attachment 1: Waves_Odd.png
Waves_Odd.png
Attachment 2: Waves_even.png
Waves_even.png
Attachment 3: WavesZ_Odd.png
WavesZ_Odd.png
Attachment 4: WavesZ_Even.png
WavesZ_Even.png
Attachment 5: EvtStas0xA.png
EvtStas0xA.png
Attachment 6: Merger0XA.png
Merger0XA.png
  119   Fri Dec 20 12:04:03 2019 NH, TDFriday 20 December 2019
Fig 1: DSSD Leakage currents ok

Fig 2: FEE Temps ok

Fig 3: aida08 has a few timestamp errors in merger

Fig 4, 5: GSI WR OK

Fig 6: All System wide checks passed 

FIg 7: Stats good

13:11 - Alpha Run Stopped, Final File: R9_6

FEEs powered off, DSSD bias off.
Attachment 1: Screenshot-CAEN_HV_Control.png
Screenshot-CAEN_HV_Control.png
Attachment 2: FeeTemps.png
FeeTemps.png
Attachment 3: mergerTSerrors.png
mergerTSerrors.png
Attachment 4: gsiwr1.png
gsiwr1.png
Attachment 5: gsiwr2.png
gsiwr2.png
Attachment 6: checks.png
checks.png
Attachment 7: stats.png
stats.png
  118   Fri Dec 20 10:19:29 2019 NHAlpha Analysis (Pre Run)
======================================                                                                                                                                                                                                                                        
AIDA Unpacker Analysis                                                                                                                                                                                                                                                        
--------------------------------------                                                                                                                                                                                                                                        
First event: Fri Dec 13 15:34:12 2019                                                                                                                                                                                                                                         
Last event : Tue Dec 17 08:52:14 2019                                                                                                                                                                                                                                         
Duration   : 321482 seconds  =  90 hours
--------------------------------------                                                                                                                                                                                                                                        
MBS Events   (M): 1748810227 (5 kHz)                                                                                                                                                                                                                                          
AIDA Words   (W): 1748072047 (5 kHz)                                                                                                                                                                                                                                          
ADC Words    (A): 785414339 (2 kHz)                                                                                                                                                                                                                                           
-- Decays    (D): 771101846 (2 kHz)                                                                                                                                                                                                                                           
-- Implants  (I):  14312493 (44 Hz)                                                                                                                                                                                                                                           
Info Words   (F): 962657708 (2 kHz)                                                                                                                                                                                                                                           
-- SYNC48    (S): 2012396012 (6 kHz)                                                                                                                                                                                                                                          
-- SYNC63    (s): 2012396012 (6 kHz)                                                                                                                                                                                                                                          
-- PAUSE     (P):   2925640 (9 Hz)                                                                                                                                                                                                                                            
-- RESUME    (R):   2925667 (9 Hz)                                                                                                                                                                                                                                            
-- Discrim   (D): 1226981673 (3 kHz)                                                                                                                                                                                                                                          
-- Unknown   (U):         0 (0 Hz)                                                                                                                                                                                                                                            
Timewarps    (T):         2 (0 Hz)
Dead Time    (X): 4336.67292 s (1.34896%)
======================================

Possibly one bond wire loss in DSSD 1
Possibly a few lost in DSSD 3 (X)
DSSD 2 is OK.

Will update with post run data for additional statistics
Attachment 1: Alphas.png
Alphas.png
Attachment 2: Alphas_E.png
Alphas_E.png
  116   Thu Dec 19 14:45:42 2019 TDFEE64 ASIC settings - different diode link settings
[npg@aidas-gsi 2019Oct31-13.24.23]$ pwd
/MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23
[npg@aidas-gsi 2019Oct31-13.24.23]$ /bin/csh
[npg@aidas-gsi 2019Oct31-13.24.23]$ foreach f (`ls`)
foreach? echo $f
foreach? cat "$f"/CONTENTS | grep .26
foreach? end
aida01
ASIC.2.26 string 0xbf
ASIC.3.26 string 0xbf
ASIC.1.26 string 0xbf
ASIC.4.26 string 0xbf
aida02
ASIC.2.26 string 0x11
ASIC.3.26 string 0x11
ASIC.1.26 string 0x11
ASIC.4.26 string 0x11
aida03
ASIC.2.26 string 0xca
ASIC.3.26 string 0xca
ASIC.1.26 string 0xca
ASIC.4.26 string 0xca
aida04
ASIC.2.26 string 0x23
ASIC.3.26 string 0x23
ASIC.1.26 string 0x23
ASIC.4.26 string 0x23
aida05
ASIC.2.26 string 0xbf
ASIC.3.26 string 0xbf
ASIC.1.26 string 0xbf
ASIC.4.26 string 0xbf
aida06
ASIC.2.26 string 0x11
ASIC.3.26 string 0x11
ASIC.1.26 string 0x11
ASIC.4.26 string 0x11
aida07
ASIC.2.26 string 0xca
ASIC.3.26 string 0xca
ASIC.1.26 string 0xca
ASIC.4.26 string 0xca
aida08
ASIC.2.26 string 0x23
ASIC.3.26 string 0x23
ASIC.1.26 string 0x23
ASIC.4.26 string 0x23
aida09
ASIC.2.26 string 0xbf
ASIC.3.26 string 0xbf
ASIC.1.26 string 0xbf
ASIC.4.26 string 0xbf
aida10
ASIC.2.26 string 0x11
ASIC.3.26 string 0x11
ASIC.1.26 string 0x11
ASIC.4.26 string 0x11
aida11
ASIC.2.26 string 0xca
ASIC.3.26 string 0xca
ASIC.1.26 string 0xca
ASIC.4.26 string 0xca
aida12
ASIC.2.26 string 0x11
ASIC.3.26 string 0x11
ASIC.1.26 string 0x11
ASIC.4.26 string 0x11

This variation appears to be true for previously saved ASIC settings too

2018Aug21-13.17.09
2018Sep06-14.38.28
2019Mar31-16.15.57
2019Oct31-13.24.23


Current ASIC settings for AIDA@RIKEN for nnaida1-nnaida4 use values 0x23 and 0xbf

aidas1> cd 2019Jun13-08.56.35
/MIDAS/DB/EXPERIMENTS/AIDA/2019Jun13-08.56.35
aidas1> ls
CONTENTS  nnaida10  nnaida12  nnaida14  nnaida16  nnaida18  nnaida2   nnaida21  nnaida23  nnaida3  nnaida5  nnaida7  nnaida9
nnaida1   nnaida11  nnaida13  nnaida15  nnaida17  nnaida19  nnaida20  nnaida22  nnaida24  nnaida4  nnaida6  nnaida8
aidas1> cat nnaida1/CONTENTS
ASIC.2.20 string 0x68
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.1.4 string 0x07
ASIC.2.21 string 0x0f
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x00
ASIC.2.26 string 0xbf
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x0d
ASIC.2.3 string 0x02
ASIC.3.12 string 0x0a
ASIC.2.4 string 0x07
ASIC.3.13 string 0x34
ASIC.2.5 string 0x08
ASIC.3.14 string 0x02
ASIC.2.6 string 0x09
ASIC.3.15 string 0x20
ASIC.2.7 string 0x05
ASIC.3.16 string 0xd2
ASIC.2.8 string 0x00
ASIC.3.17 string 0x80
ASIC.2.9 string 0x0f
ASIC.3.18 string 0xb2
ASIC.3.20 string 0x68
ASIC.3.19 string 0x5c
WAVE.0 string 0x00000000
ASIC.3.21 string 0x0f
WAVE.1 string 0x00000000
ASIC.3.22 string 0x5c
LED.0 string 0x00411ecc
WAVE.2 string 0x00000000
ASIC.3.23 string 0x08
LED.1 string 0x0000ffff
WAVE.3 string 0x00000000
ASIC.3.24 string 0x68
LED.2 string 0x00411ecc
ASIC.3.25 string 0x08
LED.3 string 0x00000000
ASIC.1.10 string 0x00
LED.4 string 0x00411ecc
ASIC.3.0 string 0x04
ASIC.1.11 string 0x0d
ASIC.3.26 string 0xbf
ASIC.1.12 string 0x0a
ASIC.3.1 string 0x05
LED.5 string 0x00000000
LED.6 string 0x00411ecc
ASIC.3.2 string 0x06
ASIC.1.13 string 0x34
LED.7 string 0x00000000
ASIC.3.3 string 0x02
ASIC.1.14 string 0x02
LED.8 string 0x000003fc
ASIC.1.15 string 0x20
ASIC.3.4 string 0x07
ASIC.3.5 string 0x08
LED.9 string 0x000000c8
ASIC.1.16 string 0xd2
ASIC.1.17 string 0x80
ASIC.3.6 string 0x09
ASIC.3.7 string 0x05
ASIC.1.18 string 0xb2
ASIC.1.19 string 0x5c
ASIC.1.20 string 0x68
ASIC.3.8 string 0x00
ASIC.3.9 string 0x0f
ASIC.1.21 string 0x0f
ASIC.1.22 string 0x5c
ASIC.4.10 string 0x00
ASIC.1.23 string 0x08
ASIC.4.11 string 0x0d
ASIC.4.12 string 0x0a
ASIC.1.24 string 0x68
ASIC.4.13 string 0x34
ASIC.1.25 string 0x08
ASIC.4.14 string 0x02
ASIC.1.26 string 0xbf
ASIC.4.15 string 0x20
DISC.6 string 0x00000002
ASIC.4.16 string 0xd2
ASIC.4.0 string 0x04
ASIC.4.17 string 0x80
DISC.7 string 0x24800000
ASIC.4.18 string 0xb2
ASIC.4.1 string 0x05
ASIC.4.2 string 0x06
ASIC.4.19 string 0x5c
ASIC.4.20 string 0x68
ASIC.4.21 string 0x0f
ASIC.4.3 string 0x02
ASIC.4.4 string 0x07
ASIC.4.22 string 0x5c
ASIC.4.23 string 0x08
ASIC.4.5 string 0x08
ASIC.4.6 string 0x09
ASIC.4.24 string 0x68
ASIC.2.10 string 0x00
ASIC.4.25 string 0x08
ASIC.4.7 string 0x05
ASIC.4.8 string 0x00
ASIC.4.26 string 0xbf
ASIC.2.11 string 0x0d
ASIC.2.12 string 0x0a
ASIC.4.9 string 0x0f
ASIC.2.13 string 0x34
ASIC.2.14 string 0x02
ASIC.2.15 string 0x20
ASIC.2.16 string 0xd2
ASIC.1.0 string 0x04
ASIC.1.1 string 0x05
ASIC.2.17 string 0x80
ASIC.2.18 string 0xb2
ASIC.1.2 string 0x06
aidas1> cat nnaida2/CONTENTS
ASIC.2.20 string 0x68
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.1.4 string 0x07
ASIC.2.21 string 0x0f
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x00
ASIC.2.26 string 0xbf
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x0d
ASIC.2.3 string 0x02
ASIC.3.12 string 0x0a
ASIC.2.4 string 0x07
ASIC.3.13 string 0x34
ASIC.2.5 string 0x08
ASIC.3.14 string 0x02
ASIC.2.6 string 0x09
ASIC.3.15 string 0x20
ASIC.2.7 string 0x05
ASIC.3.16 string 0xd2
ASIC.2.8 string 0x00
ASIC.3.17 string 0x80
ASIC.2.9 string 0x0f
ASIC.3.18 string 0xb2
ASIC.3.20 string 0x68
ASIC.3.19 string 0x5c
WAVE.0 string 0x00000008
ASIC.3.21 string 0x0f
WAVE.1 string 0x00000005
ASIC.3.22 string 0x5c
LED.0 string 0x00411ecc
WAVE.2 string 0x0000ff38
ASIC.3.23 string 0x08
WAVE.3 string 0x00000080
LED.1 string 0x0000ffff
ASIC.3.24 string 0x68
LED.2 string 0x00411ecc
ASIC.3.25 string 0x08
ASIC.1.10 string 0x00
LED.3 string 0x00000000
ASIC.1.11 string 0x0d
ASIC.3.0 string 0x04
LED.4 string 0x00411ecc
ASIC.3.26 string 0xbf
ASIC.1.12 string 0x0a
ASIC.3.1 string 0x05
LED.5 string 0x00000000
LED.6 string 0x00411ecc
ASIC.3.2 string 0x06
ASIC.1.13 string 0x34
ASIC.1.14 string 0x02
ASIC.3.3 string 0x02
LED.7 string 0x00000000
LED.8 string 0x000003fc
ASIC.1.15 string 0x20
ASIC.3.4 string 0x07
ASIC.3.5 string 0x08
ASIC.1.16 string 0xd2
LED.9 string 0x000000c8
ASIC.1.17 string 0x80
ASIC.3.6 string 0x09
ASIC.3.7 string 0x05
ASIC.1.18 string 0xb2
ASIC.1.20 string 0x68
ASIC.1.19 string 0x5c
ASIC.3.8 string 0x00
ASIC.3.9 string 0x0f
ASIC.1.21 string 0x0f
ASIC.4.10 string 0x00
ASIC.1.22 string 0x5c
ASIC.1.23 string 0x08
ASIC.4.11 string 0x0d
ASIC.4.12 string 0x0a
ASIC.1.24 string 0x68
ASIC.1.25 string 0x08
ASIC.4.13 string 0x34
ASIC.4.14 string 0x02
ASIC.1.26 string 0xbf
ASIC.4.15 string 0x20
ASIC.4.16 string 0xd2
DISC.6 string 0x00000002
ASIC.4.0 string 0x04
DISC.7 string 0x24800000
ASIC.4.17 string 0x80
ASIC.4.18 string 0xb2
ASIC.4.1 string 0x05
ASIC.4.20 string 0x68
ASIC.4.2 string 0x06
ASIC.4.19 string 0x5c
ASIC.4.3 string 0x02
ASIC.4.21 string 0x0f
ASIC.4.4 string 0x07
ASIC.4.22 string 0x5c
ASIC.4.23 string 0x08
ASIC.4.5 string 0x08
ASIC.4.6 string 0x09
ASIC.4.24 string 0x68
ASIC.2.10 string 0x00
ASIC.4.25 string 0x08
ASIC.4.7 string 0x05
ASIC.4.8 string 0x00
ASIC.4.26 string 0xbf
ASIC.2.11 string 0x0d
ASIC.2.12 string 0x0a
ASIC.4.9 string 0x0f
ASIC.2.13 string 0x34
ASIC.2.14 string 0x02
ASIC.2.15 string 0x20
ASIC.2.16 string 0xd2
ASIC.1.0 string 0x04
ASIC.1.1 string 0x05
ASIC.2.17 string 0x80
ASIC.2.18 string 0xb2
ASIC.1.2 string 0x06
aidas1> cat nnaida3/CONTENTS
ASIC.2.20 string 0x68
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.1.4 string 0x07
ASIC.2.21 string 0x0b
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x01
ASIC.2.26 string 0x23
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x02
ASIC.2.3 string 0x02
ASIC.3.12 string 0x0a
ASIC.2.4 string 0x07
ASIC.3.13 string 0xdb
ASIC.2.5 string 0x08
ASIC.3.14 string 0x02
ASIC.2.6 string 0x09
ASIC.3.15 string 0x20
ASIC.2.7 string 0x05
ASIC.3.16 string 0xd2
ASIC.2.8 string 0x01
ASIC.3.17 string 0x80
ASIC.2.9 string 0x0f
ASIC.3.18 string 0x30
ASIC.3.20 string 0x68
ASIC.3.19 string 0x5c
WAVE.0 string 0x00000008
ASIC.3.21 string 0x0b
WAVE.1 string 0x00000005
ASIC.3.22 string 0x5c
LED.0 string 0x004021fc
WAVE.2 string 0x0000ff38
ASIC.3.23 string 0x08
WAVE.3 string 0x00000080
LED.1 string 0x0000ffff
ASIC.3.24 string 0x68
LED.2 string 0x004021fc
ASIC.3.25 string 0x08
ASIC.1.10 string 0x00
LED.3 string 0x00000000
ASIC.1.11 string 0x02
ASIC.3.0 string 0x04
LED.4 string 0x004021fc
ASIC.3.26 string 0x23
ASIC.1.12 string 0x0a
ASIC.3.1 string 0x05
LED.5 string 0x00000000
LED.6 string 0x004021fc
ASIC.3.2 string 0x06
ASIC.1.13 string 0xdb
ASIC.1.14 string 0x02
ASIC.3.3 string 0x02
LED.7 string 0x00000000
LED.8 string 0x000003fc
ASIC.1.15 string 0x20
ASIC.3.4 string 0x07
ASIC.3.5 string 0x08
ASIC.1.16 string 0xd2
LED.9 string 0x000000c8
ASIC.1.17 string 0x80
ASIC.3.6 string 0x09
ASIC.3.7 string 0x05
ASIC.1.18 string 0x30
ASIC.1.20 string 0x68
ASIC.1.19 string 0x5c
ASIC.3.8 string 0x01
ASIC.3.9 string 0x0f
ASIC.1.21 string 0x0b
ASIC.4.10 string 0x00
ASIC.1.22 string 0x5c
ASIC.1.23 string 0x08
ASIC.4.11 string 0x02
ASIC.4.12 string 0x0a
ASIC.1.24 string 0x68
ASIC.1.25 string 0x08
ASIC.4.13 string 0xdb
ASIC.4.14 string 0x02
ASIC.1.26 string 0x23
ASIC.4.15 string 0x20
ASIC.4.16 string 0xd2
DISC.6 string 0x00000002
ASIC.4.0 string 0x04
DISC.7 string 0x24800000
ASIC.4.17 string 0x80
ASIC.4.18 string 0x30
ASIC.4.1 string 0x05
ASIC.4.20 string 0x68
ASIC.4.2 string 0x06
ASIC.4.19 string 0x5c
ASIC.4.3 string 0x02
ASIC.4.21 string 0x0b
ASIC.4.4 string 0x07
ASIC.4.22 string 0x5c
ASIC.4.23 string 0x08
ASIC.4.5 string 0x08
ASIC.4.6 string 0x09
ASIC.4.24 string 0x68
ASIC.2.10 string 0x00
ASIC.4.25 string 0x08
ASIC.4.7 string 0x05
ASIC.4.8 string 0x01
ASIC.4.26 string 0x23
ASIC.2.11 string 0x02
ASIC.2.12 string 0x0a
ASIC.4.9 string 0x0f
ASIC.2.13 string 0xdb
ASIC.2.14 string 0x02
ASIC.2.15 string 0x20
ASIC.2.16 string 0xd2
ASIC.1.0 string 0x04
ASIC.1.1 string 0x05
ASIC.2.17 string 0x80
ASIC.2.18 string 0x30
ASIC.1.2 string 0x06
aidas1> cat nnaida4/CONTENTS
ASIC.2.20 string 0x68
ASIC.1.3 string 0x02
ASIC.2.19 string 0x5c
ASIC.1.4 string 0x07
ASIC.2.21 string 0x0b
ASIC.2.22 string 0x5c
ASIC.1.5 string 0x08
ASIC.1.6 string 0x09
ASIC.2.23 string 0x08
ASIC.2.24 string 0x68
ASIC.1.7 string 0x05
ASIC.2.25 string 0x08
ASIC.1.8 string 0x01
ASIC.2.26 string 0x23
ASIC.1.9 string 0x0f
ASIC.2.0 string 0x04
ASIC.2.1 string 0x05
ASIC.3.10 string 0x00
ASIC.2.2 string 0x06
ASIC.3.11 string 0x02
ASIC.2.3 string 0x02
ASIC.3.12 string 0x0a
ASIC.2.4 string 0x07
ASIC.3.13 string 0xdb
ASIC.2.5 string 0x08
ASIC.3.14 string 0x02
ASIC.2.6 string 0x09
ASIC.3.15 string 0x20
ASIC.2.7 string 0x05
ASIC.3.16 string 0xd2
ASIC.2.8 string 0x01
ASIC.3.17 string 0x80
ASIC.2.9 string 0x0f
ASIC.3.18 string 0x30
ASIC.3.20 string 0x68
ASIC.3.19 string 0x5c
WAVE.0 string 0x00000008
ASIC.3.21 string 0x0b
WAVE.1 string 0x00000005
ASIC.3.22 string 0x5c
LED.0 string 0x00402134
WAVE.2 string 0x0000ff38
ASIC.3.23 string 0x08
LED.1 string 0x0000ffff
WAVE.3 string 0x00000080
ASIC.3.24 string 0x68
LED.2 string 0x00402134
ASIC.3.25 string 0x08
LED.3 string 0x00000000
ASIC.1.10 string 0x00
LED.4 string 0x00402134
ASIC.3.0 string 0x04
ASIC.1.11 string 0x02
ASIC.3.26 string 0x23
ASIC.1.12 string 0x0a
ASIC.3.1 string 0x05
LED.5 string 0x00000000
LED.6 string 0x00402134
ASIC.3.2 string 0x06
ASIC.1.13 string 0xdb
LED.7 string 0x00000000
ASIC.3.3 string 0x02
ASIC.1.14 string 0x02
ASIC.1.15 string 0x20
LED.8 string 0x000003fc
ASIC.3.4 string 0x07
ASIC.3.5 string 0x08
LED.9 string 0x000000c8
ASIC.1.16 string 0xd2
ASIC.1.17 string 0x80
ASIC.3.6 string 0x09
ASIC.3.7 string 0x05
ASIC.1.18 string 0x30
ASIC.1.19 string 0x5c
ASIC.1.20 string 0x68
ASIC.3.8 string 0x01
ASIC.3.9 string 0x0f
ASIC.1.21 string 0x0b
ASIC.4.10 string 0x00
ASIC.1.22 string 0x5c
ASIC.1.23 string 0x08
ASIC.4.11 string 0x02
ASIC.4.12 string 0x0a
ASIC.1.24 string 0x68
ASIC.1.25 string 0x08
ASIC.4.13 string 0xdb
ASIC.4.14 string 0x02
ASIC.1.26 string 0x23
ASIC.4.15 string 0x20
ASIC.4.16 string 0xd2
DISC.6 string 0x00000002
ASIC.4.0 string 0x04
DISC.7 string 0x24800000
ASIC.4.17 string 0x80
ASIC.4.18 string 0x30
ASIC.4.1 string 0x05
ASIC.4.2 string 0x06
ASIC.4.20 string 0x68
ASIC.4.19 string 0x5c
ASIC.4.21 string 0x0b
ASIC.4.3 string 0x02
ASIC.4.4 string 0x07
ASIC.4.22 string 0x5c
ASIC.4.23 string 0x08
ASIC.4.5 string 0x08
ASIC.4.6 string 0x09
ASIC.4.24 string 0x68
ASIC.2.10 string 0x00
ASIC.4.25 string 0x08
ASIC.4.7 string 0x05
ASIC.4.8 string 0x01
ASIC.4.26 string 0x23
ASIC.2.11 string 0x02
ASIC.2.12 string 0x0a
ASIC.4.9 string 0x0f
ASIC.2.13 string 0xdb
ASIC.2.14 string 0x02
ASIC.2.15 string 0x20
ASIC.2.16 string 0xd2
ASIC.1.0 string 0x04
ASIC.1.1 string 0x05
ASIC.2.17 string 0x80
ASIC.2.18 string 0x30
ASIC.1.2 string 0x06

Have modifuied AIDA@DESPEC  ASIC seetings and saved using DB key 2019Dec19-16.19.51
  115   Thu Dec 19 14:26:09 2019 TDBNC PB-5 - Manual
Attachment 1: pb-5manual.pdf
pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf pb-5manual.pdf
  114   Thu Dec 19 14:07:38 2019 TDReport - low - aida08 ADC spectrum size 32k cf. 64k channels expcted using layout
Note that aida08 spectrum 1.8.L size is (shown as) 32k channels cf. 64k channels expected using layout - attachment 1 - 5
Layout configuration - attachment 6
Why?

Current AIDA Options are shown below

[npg@aidas-gsi Options]$ pwd
/MIDAS/DB/EXPERIMENTS/AIDA/Options
[npg@aidas-gsi Options]$ csh
[npg@aidas-gsi Options]$ foreach f (`ls`)
foreach? echo $f
foreach? cat "$f"/CONTENTS
foreach? end
aida01
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
Rate.channels string 512
RunNumber string 12
Aida_GroupBase string 1
WAVE_DMA_HWM string 0x0007ffff
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
Aida_Hist_D_Enable string 1
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida02
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
da_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&WAVE_DMA_HWM&&Aida_Hist_D_Enable&&ASIC.settings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable 
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida03
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
RunNumber string 12
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida04
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Aida_GroupBase string 1
Rate.channels string 512
Stat.channels string 512
RunNumber string 12
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
MERGE.LinksAvailable string 4
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida05
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida06
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida07
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida08
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida09
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
0x0006dead string 0x0000
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida10
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
0x0006dead string 0x0000
Aida.offset string 0
MACB_TRIG_MODE string 4
Stat.channels string 512
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida11
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&WAVE_DMA_HWM&&Aida_Hist_D_Enable&&ASIC.settings&&Aida.Vchannels&&Aida.Wchannels&&Stat.shift&&Aida.channels&&Include.Aida&&Aida_Hist_V_Enable&&DataFormat&&Aida_Hist_H_Enable&&DataAcqPgm&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
MERGE.LinksInUse string 1%1%1%1%
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
RunNumber string 12
Aida_Hist_D_Enable string 1
ASIC.settings string 2019Oct31-13.24.23
Aida.Vchannels string 256
WAVE_DMA_HWM string 0x0007ffff
Aida.Wchannels string 1020
Stat.shift string 6
MERGE.LinksAvailable string 4
Include.Aida string 0
Aida_Hist_V_Enable string 0
Aida.channels string 65536
DataFormat string 0x0000
DataAcqPgm string AidaExecV9
Aida_Hist_H_Enable string 1
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
aida12
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 4
RunNumber string 12
Rate.channels string 512
Aida_GroupBase string 1
Stat.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: LayOut4.mlf
6
gallery=1 overlap=1 server=aida02 spectrum=1.8.L
gallery=2 overlap=1 server=aida04 spectrum=1.8.L
gallery=3 overlap=1 server=aida06 spectrum=1.8.L
gallery=4 overlap=1 server=aida08 spectrum=1.8.L
gallery=5 overlap=1 server=aida10 spectrum=1.8.L
gallery=6 overlap=1 server=aida12 spectrum=1.8.L
  113   Thu Dec 19 13:09:32 2019 TDMIDAS app launcher command lines
App                          Path

TclHttpd @ 8015 for AIDA     /MIDAS/Linux/startup/HTTPD
TclHttpd @ 8115 for AIDA     /MIDAS/Linux/startup/HTTPD@8115

TapeServer                   /MIDAS/Linux/startup/TapeServer
New Merger for AIDA          /MIDAS/Linux/startup/NewMerger

MBS Relay                    /MIDAS/Linux/startup/datarelaymbs
  112   Thu Dec 19 13:06:02 2019 TD, NHAIDA@DESPEC To Do list
FAIR phase 0 DESPEC experiments are scheduled March-May 2020 see

https://www.gsi.de/en/work/organisation/scientific_boards/user/beam_time.htm

The current (19.12.19) version of the schedule is v22 

https://www.gsi.de/fileadmin/beamtime/2020/BTS2020_v022_all.pdf

AIDA will used for DESPEC experiments S480, S452, S470 & S460

To be done at GSI

- check DSSSD - FEE64 connection using alpha background data collected pre- and post- Dec 19 test (NH)

- investigate system noise (TD, NH et al.)

- replace aida10 adaptor PCB and/or FEE64 (TD, NH et al.)

Other issues requiring the support of Carl, James, Patrick & Vic are 
in order of our ( = TD + NH ) priority - Carl et al. may have a different
view. If so, we should discuss.

- ASIC settings / Options DB 'corruption' issues (VFEP, CW et al.)

- waveform data (PCS, JL et al.)

- FEE64/ASIC sync (PCS, JL et al.)

ELOG V3.1.3-unknown
  111   Thu Dec 19 10:11:59 2019 TDReport - medium - FEE64 panics during boot
Some of the FEE64s aida01 .. aida12 panic during boot 

Frequencies of panics for each FEE64 can be seen in attachment 1

Below is an example of an aida04 panic following a power cycle and an automatic reboot

pi@raspberrypi:~/logs $ ./tail_aida aida04
aida04
ttyUSB7
19/11:07:04|LR [203f5680] 0x203f5680
19/11:07:04|Call Trace:
19/11:07:04|Kernel panic - not syncing: Fatal exception in interrupt
19/11:07:04|Call Trace:
19/11:07:04|[c6941de0] [c0005de8] show_stack+0x44/0x16c (unreliable)
19/11:07:04|[c6941e20] [c00345bc] panic+0x94/0x168
19/11:07:04|[c6941e70] [c000bd44] die+0x178/0x18c
19/11:07:04|[c6941e90] [c0011a28] do_page_fault+0xc4/0x458
19/11:07:04|[c6941f40] [c000e7c4] handle_page_fault+0xc/0x80
19/11:07:04|Rebooting in 180 seconds..
ISOL Version 1.00 Date 9th January 2017
Flash base address=FC000000
Set Flash to ASync Mode
XST_SUCCESS|
Finished copying zImage to RAM 
19/11:10:05|
Found 0 errors checking kernel image
19/11:10:06|VHDL version number 0X18430701 
Based on AIDA Bootloader version number 1.2.0 -- 16th August 2012
Starting LMK 3200 setup 
19/11:10:06|
Setting LMK03200 to standard clock settings -- External Clock 23Nov15
.... SPI Base Address=0x81400000
clk_control_reg=0x4
19/11:10:06|Next step is SPIconfig
Control 32(0x81400000)=0x180
SlaveSel(0x81400000)=0x3
Ctrl(0x81400000)=0xE6
Ctrl(0x81400000)=0x86
19/11:10:06|SPIconfig done now to set up the LMK3200 registers
19/11:10:06|LMK #0 : regInit[0]=0x80000000
19/11:10:07|LMK #0 : regInit[1]=0x10070600
19/11:10:07|LMK #0 : regInit[2]=0x60601
19/11:10:07|LMK #0 : regInit[3]=0x60602
19/11:10:07|LMK #0 : regInit[4]=0x60603
19/11:10:07|LMK #0 : regInit[5]=0x70624
19/11:10:07|LMK #0 : regInit[6]=0x70605
19/11:10:07|LMK #0 : regInit[7]=0x70606
19/11:10:07|LMK #0 : regInit[8]=0x70627
19/11:10:07|LMK #0 : regInit[9]=0x10000908
19/11:10:07|LMK #0 : regInit[10]=0xA0022A09
19/11:10:07|LMK #0 : regInit[11]=0x82800B
19/11:10:07|LMK #0 : regInit[12]=0x28C800D
19/11:10:07|LMK #0 : regInit[13]=0x830020E
19/11:10:07|LMK #0 : regInit[14]=0xC800180F
Calibrate completed at 941 counts
Setting Clock Control =0x0000000B, to set GOE and sync bit
Ctrl @ SPIstop (0x81400000)=0x186
Timeout waiting for Lock detect Stage 2 (Zero Delay), PWR_DWN=0x00000004
19/11:10:07|
Finished Clock setup LMK03200
completed LMK 3200 setup 
Loaded all four ASICs with default settings 
Setting the ADCs into calibration mode 
19/11:10:07|
Control 32(0x81400400)=0x180
SlaveSel(0x81400400)=0xFF
Ctrl(0x81400400)=0xE6
Ctrl(0x81400400)=0x86
Init : Config of AD9252 SPI ok
19/11:10:08|
Ctrl @ SPIstop (0x81400400)=0x186ADCs initialised 
Cal DCMs not locked 
ADC calibrate failed 
Jumping to kernel simpleboot...
19/11:10:08|
zImage starting: loaded at 0x00a00000 (sp: 0x00bc4eb0)
Allocating 0x3b78cc bytes for kernel ...
gunzipping (0x00000000 <- 0x00a0f000:0x00bc380e)...done 0x39604c bytes
19/11:10:11|
Linux/PowerPC load: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
Finalizing device tree... flat tree at 0xbd1300
Probing IIC bus for MAC... MAC address = 0xd8 0x80 0x39 0x41 0xf6 0xb7 
19/11:10:17|Using Xilinx Virtex440 machine description
19/11:10:18|Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
19/11:10:18|Zone PFN ranges:
19/11:10:18|  DMA      0x00000000 -> 0x00007000
19/11:10:18|  Normal   0x00007000 -> 0x00007000
19/11:10:18|Movable zone start PFN for each node
19/11:10:18|early_node_map[1] active PFN ranges
19/11:10:18|    0: 0x00000000 -> 0x00007000
19/11:10:18|MMU: Allocated 1088 bytes of context maps for 255 contexts
19/11:10:18|Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 28448
19/11:10:18|Kernel command line: console=ttyS0 root=/dev/nfs ip=on rw mem=112M
19/11:10:18|PID hash table entries: 512 (order: 9, 2048 bytes)
19/11:10:19|Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
19/11:10:19|Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
19/11:10:19|Memory: 109680k/114688k available (3500k kernel code, 4852k reserved, 144k data, 130k bss, 168k init)
19/11:10:19|Kernel virtual memory layout:
19/11:10:19|  * 0xffffe000..0xfffff000  : fixmap
19/11:10:19|  * 0xfde00000..0xfe000000  : consistent mem
19/11:10:19|  * 0xfde00000..0xfde00000  : early ioremap
19/11:10:19|  * 0xd1000000..0xfde00000  : vmalloc & ioremap
19/11:10:19|NR_IRQS:512
19/11:10:19|clocksource: timebase mult[a00000] shift[22] registered
19/11:10:19|Console: colour dummy device 80x25
19/11:10:19|Mount-cache hash table entries: 512
19/11:10:19|NET: Registered protocol family 16
19/11:10:19|PCI: Probing PCI hardware
19/11:10:19|bio: create slab <bio-0> at 0
19/11:10:19|NET: Registered protocol family 2
19/11:10:19|IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
19/11:10:19|TCP established hash table entries: 4096 (order: 3, 32768 bytes)
19/11:10:19|TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
19/11:10:19|TCP: Hash tables configured (established 4096 bind 4096)
19/11:10:20|TCP reno registered
19/11:10:20|NET: Registered protocol family 1
19/11:10:20|ROMFS MTD (C) 2007 Red Hat, Inc.
19/11:10:20|msgmni has been set to 214
19/11:10:20|io scheduler noop registered
19/11:10:20|io scheduler anticipatory registered
19/11:10:20|io scheduler deadline registered
19/11:10:20|io scheduler cfq registered (default)
19/11:10:20|Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
19/11:10:20|83e00000.serial: ttyS0 at MMIO 0x83e01003 (irq = 16) is a 16550
19/11:10:20|console [ttyS0] enabled
19/11:10:20|brd: module loaded
19/11:10:20|loop: module loaded
19/11:10:20|Device Tree Probing 'ethernet'
19/11:10:20|xilinx_lltemac 81c00000.ethernet: MAC address is now d8:80:39:41:f6:b7
19/11:10:20|xilinx_lltemac 81c00000.ethernet: XLlTemac: using DMA mode.
19/11:10:20|XLlTemac: DCR address: 0x80
19/11:10:20|XLlTemac: buffer descriptor size: 32768 (0x8000)
19/11:10:20|XLlTemac: Allocating DMA descriptors with kmalloc
19/11:10:20|XLlTemac: (buffer_descriptor_init) phy: 0x6938000, virt: 0xc6938000, size: 0x8000
19/11:10:20|XTemac: PHY detected at address 7.
19/11:10:20|xilinx_lltemac 81c00000.ethernet: eth0: Xilinx TEMAC at 0x81C00000 mapped to 0xD1024000, irq=17
19/11:10:21|fc000000.flash: Found 1 x16 devices at 0x0 in 16-bit bank
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21| Intel/Sharp Extended Query Table at 0x010A
19/11:10:21|Using buffer write method
19/11:10:21|cfi_cmdset_0001: Erase suspend on write enabled
19/11:10:21|cmdlinepart partition parsing not available
19/11:10:21|RedBoot partition parsing not available
19/11:10:21|Creating 5 MTD partitions on "fc000000.flash":
19/11:10:21|0x000000000000-0x000000500000 : "golden_firmware"
19/11:10:21|0x000000500000-0x000000800000 : "golden_kernel"
19/11:10:21|0x000000800000-0x000000d00000 : "user_firmware"
19/11:10:21|0x000000d00000-0x000000fe0000 : "user_kernel"
19/11:10:21|0x000000fe0000-0x000001000000 : "env_variables"
19/11:10:21|xilinx-xps-spi 81400400.hd-xps-spi: at 0x81400400 mapped to 0xD1028400, irq=20
19/11:10:21|SPI: XIlinx spi: bus number now 32766
19/11:10:21|xilinx-xps-spi 81400000.xps-spi: at 0x81400000 mapped to 0xD102C000, irq=21
19/11:10:22|SPI: XIlinx spi: bus number now 32765
19/11:10:22|mice: PS/2 mouse device common for all mice
19/11:10:22|Device Tree Probing 'i2c'
19/11:10:22| #0 at 0x81600000 mapped to 0xD1030000, irq=22
19/11:10:22|at24 0-0050: 1024 byte 24c08 EEPROM (writable)
19/11:10:22|TCP cubic registered
19/11:10:22|NET: Registered protocol family 17
19/11:10:22|RPC: Registered udp transport module.
19/11:10:22|RPC: Registered tcp transport module.
19/11:10:22|eth0: XLlTemac: Options: 0x3fa
19/11:10:22|eth0: XLlTemac: allocating interrupt 19 for dma mode tx.
19/11:10:23|eth0: XLlTemac: allocating interrupt 18 for dma mode rx.
19/11:10:23|eth0: XLlTemac: speed set to 1000Mb/s
19/11:10:25|eth0: XLlTemac: Send Threshold = 24, Receive Threshold = 4
19/11:10:25|eth0: XLlTemac: Send Wait bound = 254, Receive Wait bound = 254
19/11:10:25|Sending DHCP requests ., OK
19/11:10:26|IP-Config: Got DHCP answer from 192.168.11.99, my address is 192.168.11.4
19/11:10:26|IP-Config: Complete:
19/11:10:26|     device=eth0, addr=192.168.11.4, mask=255.255.255.0, gw=255.255.255.255,
19/11:10:26|     host=aida04, domain=dl.ac.uk, nis-domain=nuclear.physics,
19/11:10:26|     bootserver=192.168.11.99, rootserver=192.168.11.99, rootpath=/home/Embedded/XilinxLinux/ppc_4xx/rfs/aida04
19/11:10:26|Looking up port of RPC 100003/2 on 192.168.11.99
19/11:10:26|Looking up port of RPC 100005/1 on 192.168.11.99
19/11:10:26|VFS: Mounted root (nfs filesystem) on device 0:12.
19/11:10:26|Freeing unused kernel memory: 168k init
INIT: version 2.86 booting
19/11:10:27|Starting sysinit...
19/11:10:27|		Welcome to DENX & STFC Daresbury Embedded Linux Environment
19/11:10:27|		Press 'I' to enter interactive startup.
19/11:10:27|Setting clock  (utc): Thu Dec 19 10:10:28 GMT 2019 [  OK  ]
19/11:10:28|Building the cache [  OK  ]
19/11:10:28|Setting hostname aida04:  [  OK  ]
19/11:10:29|Mounting local filesystems:  [  OK  ]
19/11:10:30|Enabling /etc/fstab swaps:  [  OK  ]
19/11:10:32|Finishing sysinit...
INIT: Entering runlevel: 3
19/11:10:35|Entering non-interactive startup
19/11:10:36|FATAL: Module ipv6 not found.
19/11:10:37|Bringing up loopback interface:  [  OK  ]
19/11:10:39|FATAL: Module ipv6 not found.
19/11:10:39|Starting system logger: [  OK  ]
19/11:10:40|Starting kernel logger: [  OK  ]
19/11:10:40|Starting rpcbind: [  OK  ]
19/11:10:41|Mounting NFS filesystems:  [  OK  ]
19/11:10:42|Mounting other filesystems:  [  OK  ]
19/11:10:42|Starting xinetd: [  OK  ]
19/11:10:43|Starting midas:  Starting MIDAS Data Acquisition for aida04
19/11:10:43|xaida: device parameters: base=0x81000000 size=0x200000
19/11:10:48|Trying to free nonexistent resource <0000000081000000-00000000811fffff>
19/11:10:49|xaida: mem region start 0x81000000 for 0x200000 mapped at 0xd2100000
19/11:10:49|xaida: driver assigned major number 254
19/11:10:49|Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
19/11:10:54|AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
19/11:10:54|AIDAMEM: aidamem: driver assigned major number 253
19/11:10:54|System identified is CPU ppc; Platform is unix; OS is Linux and Version is 2.6.31
19/11:11:01|Environment selected is CPU ppc; Platform unix; OS Linux and Operating System linux-ppc_4xx
19/11:11:01|MIDASBASE = /MIDAS and MIDAS_LIBRARY = /MIDAS/TclHttpd/linux-ppc_4xx
19/11:11:01|PATH = /MIDAS/bin_linux-ppc_4xx:/MIDAS/TclHttpd/linux-ppc_4xx:/MIDAS/linux-ppc_4xx/bin:/MIDAS/linux-ppc_4xx/bin:/sbin:/usr/sbin:/bin:/usr/bin
19/11:11:01|Computer Name = aida04; Temp Directory = /tmp/tcl361
19/11:11:05|package limit is not available: can't find package limit
19/11:11:07|Running with default file descriptor limit
19/11:11:07|package setuid is not available: can't find package setuid
19/11:11:09|Running as user 0 group 0
19/11:11:09|
19/11:11:10|AIDA Data Acquisition Program Release 9_10.Apr  3 2019_11:34:31 starting
19/11:11:10|
19/11:11:10|Built without pthreaxaida: open:
19/11:11:10|ds
19/11:11:10|
19/11:11:10|Creating NetAIDAMEM: aidamem_open:
19/11:11:10|Vars
19/11:11:10|Output buffer length = 65504; format option = 4; transfer option = 3
19/11:11:11|EB transfer option = 3
19/11:11:11|NetVars created and initialised
19/11:11:11|Statistics thread starting
19/11:11:11|Statistics thread created
19/11:11:11|Stat/Rate creation thread starting
19/11:11:11|Data Acquisition task has PID 375
19/11:11:11|Stat/Rate creation thread created
19/11:11:11|Hit/Rate creation thread starting
19/11:11:11|Hit/Rate creation thread created
19/11:11:11|AIDA Heartbeat thread starting
19/11:11:11|Heartbeat thread created
19/11:11:11|Installing signal handlers
19/11:11:11|Done
19/11:11:11|ModuleNum = 0
19/11:11:11|Aida Initialise complete. AidaExecV9_10: Build Apr  3 2019_11:34:31. HDL version : 18430701 
19/11:11:11|Spectra table initialised
19/11:11:11|AIDA Data Acquisition now all ready to start
19/11:11:11|SIGBUS, SIGSEGV and SIGPIPE traps setup
19/11:11:11|/debug user "debug" password "-f9x7ruru8cg"
19/11:11:17|httpd started on port 8015
19/11:11:18|
19/11:11:18|Cannot use /MIDAS/config/TclHttpd/aida04@8015/startup.tcl
19/11:11:18|Custom startup from /MIDAS/config/TclHttpd/aida04/startup.tcl
19/11:11:18|XAIDA Access package 1.0
19/11:11:19|/XAIDAAccessServer
19/11:11:19|XAD9252 Access package 1.0
19/11:11:20|/XAD9252AccessServer
19/11:11:20|/DataBaseAccessServer
19/11:11:20|/NetVarService
19/11:11:20|/SigTaskService
19/11:11:20|Loaded MemSasAccess
19/11:11:20|/SpectrumService
19/11:11:20|loading tcl/AIDARunControl.tcl for namespace ::
19/11:11:20|/DataAcquisitionControlServer
19/11:11:20|DefineMessage unknown
19/11:11:20|Run Control Server Implementation for AIDA
19/11:11:21|RunControlServer loaded
19/11:11:21|loading Html/RunControl/implementation.tcl
19/11:11:21|[  OK  ]
19/11:11:21|/MIDAS/TclHttpd/Html/RunControl/common.tcl returned z=1 and couldn't read file "/MIDAS/TclHttpd/Html/RunControl/common.tcl": no such file or directory
19/11:11:21|ReadRegister failed: Name=NetVar.EXEC.ID; Code= 0x10004; Info= Register name does not exist
19/11:11:21|
19/11:11:21|DENX ELDK version 4.2 build 2008-04-01
19/11:11:21|Linux 2.6.31 on a ppc
19/11:11:21|
19/11:11:21|aida04 login: Created UI registers
19/11:11:22|RunControl loaded
19/11:11:22|loading Html/AIDA/RunControl/implementation.tcl for namespace ::
19/11:11:22|AIDA RunControl loaded
19/11:11:24|Completed custom startup from /MIDAS/TclHttpd/Html/AIDA/RunControl/stats.defn.tcl
Attachment 1: 300.png
300.png
  110   Thu Dec 19 09:22:52 2019 TDThusrday 19 December
10.15 Found DAQ running at c. 1M data items/s 

      Rate spectra show aida09 1.*.H and one 1x LEC channel of aida08 asic running hot

      Check ASIC settings *all* FEE64s *all* ASICs restores rates to normal


      /TapeData/171219/R2_* files indicates files R2_0 ... R2_1 are probably OK and
      problem started c. 08.00 - activity in S4?

-rw-rw-r--. 1 npg npg 2097152000 Dec 19 01:56 R2_0
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 05:30 R2_1
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:05 R2_2
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:15 R2_3
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:29 R2_4
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:34 R2_5
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:38 R2_6
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:43 R2_7
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:47 R2_8
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:52 R2_9
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 08:56 R2_10
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:00 R2_11
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:05 R2_12
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:09 R2_13
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:14 R2_14
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:18 R2_15
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:23 R2_16
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:27 R2_17
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:31 R2_18
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:36 R2_19
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:40 R2_20
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:45 R2_21
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:49 R2_22
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:54 R2_23
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 09:58 R2_24
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:02 R2_25
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:07 R2_26
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:11 R2_27
-rw-rw-r--. 1 npg npg 2097152000 Dec 19 10:16 R2_28
-rw-rw-r--. 1 npg npg 1582563328 Dec 19 10:19 R2_29

10.31 1.8.L and Stat spectra - attachments 1-3

10.32 DAQ stop file 171219/R2_30

      zero *all* histograms

      Check ASIC settings *all* FEE64s *all* ASICs


10.48 system wide checks OK except aida07 calibration & master clock status fail - attachments 4-9

      GSI WR timestamp control OK - attachments 10-11

      DSSSD biases & leakage currents OK - attachment 12
      FEE64 temperatures OK - attachment 13
      Good event statistics OK - attachment 14

11.31 DAQ/Merger/TapeServer stopped merging/writing data at start of new run

      Stop DAQ/Merger/TapeServer, reboot FEE64s and restart from scratch

      DAQ start file 171219/R8

      slow comparator 0x64

      alpha background

      data rate c. 170kb/s

15.30 DAQ continues OK file 171219/R9_0
    
      DSSSD biases & leakage currents OK - attachment 15
      BNC Pb-5 settings - attachment 16
      FEE64 temperatures OK - attachment 17
      Good event statistics OK - attachment 18

      system wide checks OK except master clock status fail
  
      GSI WR timestamp control OK - attachments 19-20

      Merger & server consoles - attachment 21
Attachment 1: 100.png
100.png
Attachment 2: 101.png
101.png
Attachment 3: 102.png
102.png
Attachment 4: 110.png
110.png
Attachment 5: 111.png
111.png
Attachment 6: 112.png
112.png
Attachment 7: 113.png
113.png
Attachment 8: 114.png
114.png
Attachment 9: 115.png
115.png
Attachment 10: 116.png
116.png
Attachment 11: 117.png
117.png
Attachment 12: 200.png
200.png
Attachment 13: 201.png
201.png
Attachment 14: 202.png
202.png
Attachment 15: 1.png
1.png
Attachment 16: 2.png
2.png
Attachment 17: 3.png
3.png
Attachment 18: 4.png
4.png
Attachment 19: 5.png
5.png
Attachment 20: 6.png
6.png
Attachment 21: 8.png
8.png
  109   Wed Dec 18 17:39:27 2019 TDWednesday 18 December
18.40 Detector biases & leakage currents OK - attachment 1
      FEE64 temperatures OK - attachment 2
      FEE64 good event statistics OK - attachment 3
      3x <20k, 8x <50k, 11x <100k
      note - all 12x FEE64s zero disc-info #6

      system wide checks - see attachments 4-9
      master clock status fail
      WR decoder status - aida08 fails

18.49 GSI WR status control & timestamps - attachments 10 & 11

18.50 check ASIC control *all* FEE64s *all* ASICs

18.58 Rate, Stat, 1.8.L and 1.8.H spectra all FEE64s - attachments 12-17
      OK except aida10 no HEC data & low amplitude LEC data

19.00 *all* histograms zero'd

19.03 NewMerger, TapeServer and full screen - attachments 18-20

19.07 Options, ASIC settings aida09 & aida10 - attachments 21-23

20.15 collect all WR status erro counters for baseline
      WR decoder status - no errors

21.56 FRS DAQ crashed and not currently recoverable
      Stop current run and run background alphas overnight

21.57 run stopped file 171219/R1_304

      slow comparator threshold 0xf -> 0x64
      check ASIC control *all* FEE64s *all* ASICs

      BNC PB-5 attenuation x1 -> x1000

      *all* histograms zero'd

22.02 DAQ start file 171219/R2

      background alphas



 
Attachment 1: 1.png
1.png
Attachment 2: 2.png
2.png
Attachment 3: 3.png
3.png
Attachment 4: 4.png
4.png
Attachment 5: 5.png
5.png
Attachment 6: 6.png
6.png
Attachment 7: 7.png
7.png
Attachment 8: 8.png
8.png
Attachment 9: 9.png
9.png
Attachment 10: 10.png
10.png
Attachment 11: 11.png
11.png
Attachment 12: 21.png
21.png
Attachment 13: 22.png
22.png
Attachment 14: 23.png
23.png
Attachment 15: 24.png
24.png
Attachment 16: 25.png
25.png
Attachment 17: 26.png
26.png
Attachment 18: 30.png
30.png
Attachment 19: 31.png
31.png
Attachment 20: 32.png
32.png
Attachment 21: 40.png
40.png
Attachment 22: 41.png
41.png
Attachment 23: 42.png
42.png
  108   Wed Dec 18 02:44:27 2019 NH17.12.19, 18.12.19
Beam: 238U (850 MeV/u)
Fragment: 168Re

Tuning accelerator and FRS, believed since 3AM we are implanting on AIDA.

FEE issues:
One HEC in aida03 seems to be empty - LEC channel fills
aida09 has a lot of weird channels, check/load doesn't seem to fix
aida10 still no data after cable replaced

DSSD leakages OK - slightly higher now

Temps good


Fig 4: Rates 14:32

Leakages and temps good AS Before

Checking if we are implanting properly - possibly into the first DSSD at the moment

Fig 5 FRS PID
Fig 6 DSSD 
Fig 7 DSSD (PID gated)
Attachment 1: 1812-stats.png
1812-stats.png
Attachment 2: 1812-leakages.png
1812-leakages.png
Attachment 3: 1812-temps.png
1812-temps.png
Attachment 4: 1812v2-Rates.png
1812v2-Rates.png
Attachment 5: nh_1712_zaoq.png
nh_1712_zaoq.png
Attachment 6: nh_1712_dssd.png
nh_1712_dssd.png
Attachment 7: nh_1712_dssd_id.png
nh_1712_dssd_id.png
  107   Mon Dec 16 14:48:02 2019 NH TDAIDA 16.12.19
DSSD Snout was reassembled to install the second bPlast (downstream) 
 (fig 1)

Ribbon cable for aida10 was replaced (DSSD3 upper)

All DSSDs bias again (see fig 2)

Alpha run in progress - channels OK with 2 exceptions  (fig 3)

- aida10 still recording minimal to no counts

- channel 55 aida08 reading lots of data even with comparator 0x64 - corresponds to DSSD1 strip 0 (bottom strip)
  possible ground/cable issue? will check when access possible (fig 4)

Unusual MIDAS error noticed (fig 5) - restart of HTTPd fixed it
Attachment 1: 20191213_103710.jpg
20191213_103710.jpg
Attachment 2: 161219_aida_DSSDs.png
161219_aida_DSSDs.png
Attachment 3: 161219_rates.png
161219_rates.png
Attachment 4: 161219_aida0855.png
161219_aida0855.png
Attachment 5: midasmessage131219.png
midasmessage131219.png
  106   Wed Dec 11 10:58:52 2019 NHInitital Dec. Analysis
Fig 1: 2D hit pattern of each AIDA DSSD for MBS run 24 (40Ar primary beam, centred)

Low statistics in DSSD2 partly due to low energy deposition 

Fig 2: Position according to TPC42 (ca. 2 metres before AIDA)

Fig 3: 2D hit pattern of DSSD 1 for MBS run 37 (34Si fragments, no PID gate)

-
Bottom part of AIDA noticably cut off - rotation in DSSD also noticed (not an issue)

Snout was investigated and platform was discovered to not be fully level by Bogdan using a laser level - he has adjusted it so that the AIDA snout is now level to within 1 mm.
(The beam right side was previous 4-6mm below the beam left side)

Y cut in fragment run is currently of unknown origin. Optics are being investigated

-

Fig 4: Hit pattern of 40Ar with strip offsets fixed (no missing strips)

Fig 5: Energy loss in DSSD1 of 40Ar primary beam

Fig 6: TPC 41, TPC 42, AIDA X comparison

Fig 7: " Y comparison

Fig 8-11: As above for 34Si run

Y cut unknown still.


 
Statistics of 40Ar file
======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Fri Dec  6 05:21:42 2019
Last event : Fri Dec  6 05:26:17 2019
Duration   : 274.604 seconds
--------------------------------------
MBS Events   (M): 131808426 (481 kHz)
AIDA Words   (W): 110608065 (403 kHz)
ADC Words    (A): 36869349 (134 kHz)
ADC Decays   (D): 36782621 (134 kHz)
ADC Implants (I):    86728 (316 Hz)
Info Words   (F): 73738716 (269 kHz)
SYNC48 Words (S): 36869354 (134 kHz)
SYNC63 Words (s): 36869354 (134 kHz)
PAUSE Words  (P):        4 (0 Hz)
RESUME Words (R):        4 (0 Hz)
Discrim Words(D):        0 (0 Hz)
Unknown Words(U):        0 (0 Hz)
Timewarps    (T):        0 (0 Hz)
Dead Time    (X): 0.113631 s (0.0413799%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |        W        A        D        I        F        S        s    P    R    D    U    T        X
   1 |  4383018  1461006  1439401    21605  2922012  1461006  1461006    0    0    0    0    0 0
   2 | 11468351  3822783  3814577     8206  7645568  3822783  3822783    1    1    0    0    0 0.00733666
   3 |  4353624  1451208  1428500    22708  2902416  1451208  1451208    0    0    0    0    0 0
   4 |  8355002  2785000  2755870    29130  5570002  2785001  2785001    0    0    0    0    0 0
   5 |  1133544   377848   376489     1359   755696   377848   377848    0    0    0    0    0 0
   6 |  9617940  3205980  3205767      213  6411960  3205980  3205980    0    0    0    0    0 0
   7 |  4385477  1461825  1460962      863  2923652  1461826  1461826    0    0    0    0    0 0
   8 | 15105135  5035043  5032985     2058 10070092  5035045  5035045    1    1    0    0    0 0.0126429
   9 | 37487211 12495735 12495422      313 24991476 12495736 12495736    2    2    0    0    0 0.0936512
  10 |   110316    36772    36772        0    73544    36772    36772    0    0    0    0    0 0
  11 | 12329904  4109968  4109763      205  8219936  4109968  4109968    0    0    0    0    0 0
  12 |  1878543   626181   626113       68  1252362   626181   626181    0    0    0    0    0 0


Statistics of 34Si file
======================================
AIDA Unpacker Analysis
--------------------------------------
First event: Sat Dec  7 05:20:22 2019
Last event : Sat Dec  7 05:23:51 2019
Duration   : 208.635 seconds
--------------------------------------
MBS Events   (M): 148639738 (714 kHz)
AIDA Words   (W): 132033086 (634 kHz)
ADC Words    (A): 44011024 (211 kHz)
ADC Decays   (D): 43874168 (210 kHz)
ADC Implants (I):   136856 (657 Hz)
Info Words   (F): 88022062 (423 kHz)
SYNC48 Words (S): 44011028 (211 kHz)
SYNC63 Words (s): 44011028 (211 kHz)
PAUSE Words  (P):        3 (0 Hz)
RESUME Words (R):        3 (0 Hz)
Discrim Words(D):        0 (0 Hz)
Unknown Words(U):        0 (0 Hz)
Timewarps    (T):        0 (0 Hz)
Dead Time    (X): 0.0806864 s (0.0386734%)
======================================
======================================
Individual FEE Overview
======================================
 FEE |        W        A        D        I        F        S        s    P    R    D    U    T        X
   1 |  6814953  2271651  2255653    15998  4543302  2271651  2271651    0    0    0    0    0 0
   2 | 14012582  4670860  4668280     2580  9341722  4670860  4670860    1    1    0    0    0 0.0088226
   3 |  7618290  2539430  2523614    15816  5078860  2539430  2539430    0    0    0    0    0 0
   4 |  8928934  2976310  2950380    25930  5952624  2976312  2976312    0    0    0    0    0 0
   5 |  1631415   543805   531957    11848  1087610   543805   543805    0    0    0    0    0 0
   6 | 10908021  3636007  3633830     2177  7272014  3636007  3636007    0    0    0    0    0 0
   7 |  5644296  1881432  1864053    17379  3762864  1881432  1881432    0    0    0    0    0 0
   8 | 14863755  4954585  4931620    22965  9909170  4954585  4954585    0    0    0    0    0 0
   9 | 36745143 12248379 12244135     4244 24496764 12248380 12248380    2    2    0    0    0 0.0718638
  10 |   198573    66191    66191        0   132382    66191    66191    0    0    0    0    0 0
  11 | 15101900  5033966  5018830    15136 10067934  5033967  5033967    0    0    0    0    0 0
  12 |  9565224  3188408  3185625     2783  6376816  3188408  3188408    0    0    0    0    0 0

All good (no deadtime issues)
Attachment 1: AIDA_HitDec19.png
AIDA_HitDec19.png
Attachment 2: TPCs.png
TPCs.png
Attachment 3: AIDA-fragments.png
AIDA-fragments.png
Attachment 4: AIDA_XY.png
AIDA_XY.png
Attachment 5: AIDA_E.png
AIDA_E.png
Attachment 6: TPC_41_42_AIDA_X.png
TPC_41_42_AIDA_X.png
Attachment 7: TPC_41_42_AIDA_Y.png
TPC_41_42_AIDA_Y.png
Attachment 8: AIDA_XY_34Si.png
AIDA_XY_34Si.png
Attachment 9: AIDA_E_34Si.png
AIDA_E_34Si.png
Attachment 10: TPC_41_42_AIDA_X_34Si.png
TPC_41_42_AIDA_X_34Si.png
Attachment 11: TPC_41_42_AIDA_Y_34Si.png
TPC_41_42_AIDA_Y_34Si.png
  105   Sat Dec 7 02:39:02 2019 DK, NH et al.34Si Fragments PID
See attached

Tentative PID (DK) as attachment #3

DESPEC LISE++ file as attachment #4
Attachment 1: 2019-12-07-PID.png
2019-12-07-PID.png
Attachment 2: 59_AM.png
59_AM.png
Attachment 3: pid.png
pid.png
Attachment 4: Ar-300_S4_updated_matter_DEPSEC_34Si.lpp
Version 11.0.88

{============================= Main Part ======================================}
[general]
                    File = C:\Users\Nic\Documents\Ar-300_S4_updated matter_DEPSEC 34Si.lpp
                    Date = 06-12-2019
                    Time = 23:05:23
           Configuration = GSI\FRS-TA2-S4_2014.lcn
             Optionsfile = GSI_FRS_2012.lopt
                   Title = GSI FRS - TA2 -2014
          BlockStructure = MMMDSWMDMSAMMMMMMWMMDMSDMMMMMMMMMMMMMMMMMMSMMMWMMMMMMMMMMMMMMMMMM

[settings]
                   A,Z,Q = 40Ar18+                               ; Mass ElementName Charge+ Beam
                  Energy = 300                  MeV/u            
               Intensity = 1e+7                 pps              ; enA,pna,pps,kW
            RF frequency = 20                   MHz              
            Bunch length = 1                    ns               
         Settings on A,Z = 34Si                                  ; Mass ElementName Charge+ Beam

[OpticsBeam]
                      BX = 1                    (±)mm            ; one-half the horisontal beam extent (x)
                      BT = 5                    (±)mrad          ; one-half the horisontal beam divergence(x')
                      BY = 1.5                  (±)mm            ; one-half the vertical beam extent (y)
                      BF = 3.33                 (±)mrad          ; one-half the vertical beam divergence (y')
                      BL = 0                    (±)mm            
                      BD = 0.05                 (±)%             ; one-half of the momentum spread (dp/p)
                  ShiftX = 0                    mm               ; beam respect to the spectrometer axis
                  AngleX = 0                    mrad             ; beam respect to the spectrometer axis
                  ShiftY = 0                    mm               ; beam respect to the spectrometer axis
                  AngleY = 0                    mrad             ; beam respect to the spectrometer axis
            Scheme Angle = 0                    degrees          
                  ShapeX = 1
                  ShapeT = 1
                  ShapeY = 1
                  ShapeF = 1
                  ShapeL = 1
                  ShapeD = 1
               OptBeam_X = 1                    (±)mm            
               OptBeam_T = 30                   (±)mrad          
               OptBeam_Y = 1                    (±)mm            
               OptBeam_F = 30                   (±)mrad          
               OptBeam_L = 0                    (±)mm            
               OptBeam_D = 1.5                  (±)%             

[options]
               NP simple = 32                                    ; Number of points in distribution
        NP charge states = 32                                    ; Number of points in distribution
                NP wedge = 32                                    ; Number of points in distribution
           Charge states = No                                    ; No  & Yes
           CutEdgeEffect = 1                                     ; 1-Yes. Default, 0-no - for extended configurations
       Prim.beam scatter = 0                                     ; 0-without, 1-with
              Delta peak = 0                                     ; 0-without, 1-with
             BrhoMeanMax = 1                                     ; 0-Max, 1-Mean
            BrhoMeMaLeRi = 3                                     ; 0-Max, 1-Mean, 2-Left, 3-Right /for fission/

[target]
         Target contents = 0,4,1,9.012                           ; Nomer,Z,Atoms,Mass
        Target thickness = 1,2500.7,1.848,0,0,0                  ; State,Thickness,density,angle,SolidGas,..
  Target fusion compound = 0
   Targ use for Q-states = 1
           Target Defect = 1,0                                   ; [0] choice - % or micron at 0 degree,  [1]=value; 
       Degrader contents = 0,22,1,47.9                           ; Nomer,Z,Atoms,Mass
      Degrader thickness = 0,35,4.519,0,0,0                      ; State,Thickness,density,angle,SolidGas,..
  Degra use for Q-states = 1
         Degrader Defect = 1,0                                   ; [0] choice - % or micron at 0 degree,  [1]=value; 

[mechanism]
                Reaction = 0                                     ; 0 - fragm, 1 - fusion-resid, 2 - fusion-fission
               CalcOther = 1                                     ; calculate other reactions
           V calculation = 5                                     ; 0 - constant, 1 - Borrel, 2 - Rami, 3-convolution, 4-two body reaction
             V_opt/Vbeam = 1                                     ; default 1
         Velocity_exceed = 1                                     ; 0 - without, 1-with - two-body recations velocity corrections
Binding Energy for Vf/Vp = 8                    MeV              ; Binding energy for Borrel's expression
    Shift for Vf/Vp calc = 0     
        Prefragment_Rami = 1                                     ; 1-Yes, 0-No
                  Sigma0 = 90                   MeV/c            ; default 90
                  SigmaD = 0                    MeV/c            ; default 200
                  SigmaM = 87                   MeV/c            ; default 87
               Asymmetry = 0                    %                ; default 0
          Method v-sigma = 0                                     ; 0 - Goldhaber, 1-Morrissey,2-Friedman,3-Convolution
               G_Surface = 0.95                 MeV/fm^2         
 Symmetry around half_Ab = 1                                     ; 1 - yes, 0-no
 Pfaff pickup correction = 0                                     ; 1 - yes, 0-no
     ChargeExchangePfaff = 0                                     ; 1 - exclude, 0-forget
            Sigma corr 0 = 0                                     ; Coulomb energy
            Sigma corr 1 = 0                                     ; Projectile mass
           Friedman mode = 2                                     ; 0-Qgg, 1-Surface, 2-Qgg+Surface
         Prefragment_Fri = 1                                     ; 1-Yes, 0-No
        Coulomb_Friedman = 1                                     ; 1-Yes, 0-No
                  K_Morr = 8                    MeV/A            ; E/A=8MeV/A default; D.Morrissey coef.
               K_MorHalf = 8                    MeV/A            ; E/A=8MeV/A default at Afrag=Aproj/2; D.Morrissey coef.
                 AA_fast = 0                                     ; 1-Yes, 0-No
            BarrierShape = 1                                     ; 0-classical, 1-quantum mech.
                 H_omega = 5                    MeV              ; default 3
           Probabilty_CN = 0                                     ; 0/1 use Prbabilty for CN formation
            UseVanishing = 1                                     ; 1-Yes, 0-No
              VanishMode = 0                                     ; 0-Sierk, 1-Cohen
           NuclPotential = 1                                     ; 0-Bass, 1-WS
                   WS_V0 = 105                  MeV              
                   WS_R0 = 1.12                 fm               
                    WS_a = 0.75                 fm               
          FusDiffuseness = 1    
              Width Coef = 1                                     ; default 1; for Leon's charge state distribution
          gZt Correction = 1                                     ; default 1;      Leon's C.S.D.
           PowerCoefLeon = 0.477                                 ; default 0.477; Leon's C.S.D.
           Cross section = File                                  ; Fit  & File
           Charge method = 3                                     ; charge calculations method  0-5
      EPAX Cross Section = 4                                     ; cross section calculations method 0-4
        SR Cross Section = 2                                     ; EPAX for SR  0-2
             Energy Loss = 2                                     ; energy loss calculation method 0-3
     Anglular straggling = 1                                     ; 0-LISE, 1-ATIMA
         StragglingCoef1 = 0.217
         StragglingCoef2 = 1.12 
       Energy straggling = 1                                     ; 0-LISE, 1-ATIMA
       EnergyStragMethod = 0                                     ; 0-integrate, 1-table
        EnergyStragShape = 0                                     ; 0-Gauss, 1-Landau-Vavilov
          EquilThickness = 1                                     ; 0-Charge, 1-Global
              MassMethod = 0                                     ; 0-DB+calcul, 1 + just calcul
            MassDataBase = 0                                     ; 0-A&W, 1-User ME
            Mass formula = 2                                     ; 0-LDM, 1-Myerer, 2: 1+corrections
      UseChargeForEnergy = 2                                     ; 0-No, 1-Yes, 2-Auto
         EnergyValueAuto = 30                                    ; default value 30 MeV/u
         EquilibriumMode = 1                                     ; 0-Equil, 1-NonEquil
               UB_Global = 70                                    ; default 70 MeV/u
             MinZ_Global = 2                                     ; default Z>=29
        ChargeStateOptim = 1                                     ; 0-No, 1-Yes
         ZmQ_AfterReactn = 0                                     ; default 0 (full stripped)
             EPAX_p_Norm = 1
               EPAX_p_Un = 1.65
              EPAX_p_Up0 = 1.79
              EPAX_p_Up1 = 0.00472
              EPAX_p_Up2 = -1.3e-5 
                EPAX_p_H = 1

[fission]
         FisAngDistShape = 0                                     ; 0-isotropic; 1-anisotropic
     FisMomCutForAngDist = 2                                     ; 0-dont use;  1-use just MatrixKinematics; 2-use for all; (default 2)
      OddEvenCorrections = 1                                     ; 0-dont use;  1-use
 PostScissionEvaporation = 1                                     ; 0-dont use;  1-use
   DeexcitFunctionPoints = 0                                     ; 0- average deexcitation energy;  1- 3 points; 2 - manually
           FisEXmanually = 20                                    ; Excitation energy manually
           FisCSmanually = 1000                                  ; Cross section manually
            FisTXEmethod = 1                                     ; 0-from Edissipated, 1 from Q-value
                   Fis_f = 0.0045                                ; default 0.0045
              FisEXsigma = 5.5                  MeV              ; default 5.5
            FisCS_Global = 1e-11    
               FisCS_TKE = 1e-8     
                FisBeta1 = 0.625                                 ; deformation of light fragment
                FisBeta2 = 0.625                                 ; deformation of heavy fragment
                FisTKE_d = 2                    fm               ; d-param in Wilkins formula
              FisBetaFit = 1                                     ; 0-manual, 1-fit
                      N0 = 83                                    ; default 82
                     dU0 = -2.65                                 ; default -2.5
                      C0 = 0.7                                   ; default 1.4
                   cpol0 = 0.65                                  ; default 0.65
                  width0 = 0.63                                  ; default 0.63
                      N1 = 90                                    ; default 90
                     dU1 = -3.8                                  ; default -5.5
                      C1 = 0.15                                  ; default 0.16
                   cpol1 = 0.55                                  ; default 0.55
                  width1 = 0.97                                  ; default 0.97

[charge_suppression]
                 FragInd = 1e-3     
               FragTotal = 1e-5     
                 BeamInd = 1e-20    
               BeamTotal = 1e-20    

[convolution]
        Convolution mode = 1                                     ; 0-Qgg, 1-Surface, 2-Qgg+Surface
               SigmaConv = 91.5                 MeV/c            ; default 90 for Convolution
              CoefConv_0 = 3.344 
              CoefConv_1 = 3     
              CoefConv_2 = 2.936 
             ShiftConv_0 = 0.158 
             ShiftConv_1 = 0.149 
             ShiftConv_2 = 0.153 

[evaporation]
          NP evaporation = 32                                    ; Number of points in distribution
              EvapMethod = 2
        StateDensityMode = 2                                     ; 0, 1+pairing, 2+shell
      EvapUnstableNuclei = 1                                     ; 0 - only stable,1 +unstable
              Tunnelling = 1                                     ; 1-Yes, 0-No
          AvoidUnboundCS = 1                                     ; 1-Yes, 0-No
       ProtectedChannels = 1                                     ; 1-Yes, 0-No
           R_Evaporation = 5.7                  fm               ; correction for the effective Coulomb barrier
         Mode_Apf_manual = 0                                     ; 1-manual, 0-auto
             Energy_in_T = 2                                     ; default 2
     EvaporationVelocity = 0                                     ; 0 - quality, 1 -fast
        DeltaOddEvenEvap = 12    
     DeltaOddEvenFission = 14    
   BreakupTemperature250 = 4.7  
   BreakupTemperature150 = 5.9  
   BreakupTemperature050 = 8    
      BreakupDiffuseness = 0.05 
      DissipationKramers = 0                                     ; 0 - no, 1 - use
 DissipationStepFunction = 1                                     ; 0 - no, 1 - use
         DissipationBeta = 1                                     ; default 2.0
                 mode_1n = 1                                     ; 1-Yes, 0-No
                 mode_2n = 0                                     ; 1-Yes, 0-No
                 mode_1p = 1                                     ; 1-Yes, 0-No
                 mode_2p = 0                                     ; 1-Yes, 0-No
                  mode_a = 1                                     ; 1-Yes, 0-No
                  mode_d = 0                                     ; 1-Yes, 0-No
                  mode_t = 0                                     ; 1-Yes, 0-No
                mode_3he = 0                                     ; 1-Yes, 0-No
                mode_fis = 1                                     ; 1-Yes, 0-No
             mode_brk_up = 1                                     ; 1-Yes, 0-No
              mode_gamma = 0                                     ; 1-Yes, 0-No

[fission_barrier]
    FissionBarrierFactor = 1    
      FissionBarrierMode = 1                                     ; #0-4
      OddEvenCorrections = 1                                     ; 1-Yes, 0-No
        ShellCorrections = 1                                     ; 1-Yes, 0-No
             FB_InOutMax = 2                                     ; #0-2 - in/out/max
             ModeForUser = 1                                     ; #0-2
           NdeltaOddEven = 2.5  
           ZdeltaOddEven = 9    

[excitation_energy]
       GeomAA_Correction = 1                                     ; 0 - don't use,1 - use -default
          Thermalization = 0                                     ; 1-Yes, 0-No
          ThermaTimeCoef = 3e+0                                  ; 2.1e-22 MeV *s/e(t)
                Friction = 0                                     ; 0 - off,1 - on
          Ev_A_SigmaCoef = 9.6  
         G_FrictionCoef1 = 6.5  
         G_FrictionCoef2 = 0    
           G_FactorCoef1 = 1.5  
           G_FactorCoef2 = 2.5  
               DepthHole = 40    
          EnergyCoef_CB0 = 0       
          EnergyCoef_CB1 = 13.3     
          EnergyCoef_CB2 = 0       
           SigmaCoef_CB0 = 0       
           SigmaCoef_CB1 = 9.6     
           SigmaCoef_CB2 = 0       
              D_MeanTemp = 13       
               AA_factor = 1       
          ApplyLimitTemp = 0                                     ; 1-Yes, 0-No

[evapauto]
                  tun_a0 = -0.61392
                  tun_a1 = 0.44559
                  tun_a2 = 0.12008
                 A_Bound = 300                                   ; mass
               A_Pairing =  70                                   ; mass

[plot]
            Start target = Detector                              ; Detector & RF      
            Start of TOF = M7
            Stop  of TOF = M34
         dE-detector-1st = M28
         dE-detector-2nd = M7
            TKE-detector = M41
                PlotBrho = D1
               PlotWedge = S4
              X-detector = M19
              Y-detector = M19
                 Tilting = M1
                 Stopper = W1
                RO_Wedge = n
          ConditionBlock = A10
          Plot threshold = 1e-10                pps              ; minimal value for plot scale
     Shift of TOF for RF = 0                    ns               ; for dE-TOF plot with RF
  Fraction of RF trigger = 2
            UseCondition = 0
         TKE_calibration = 1,1,0,MeV                             ; Input PV(0) or CH(1), A, B, dimension

[cs_file]
              UserDiffCS = 0                                     ; Number of User Diff CS saved in this file
         AppendOverwrite = 1
          AttachedInside = 1
            ShowCSinPlot = 1
                    Chi2 = 1
              CSfilename = 

[sec_reactions]
        NP sec.reactions = 16                                    ; Number of points in distribution
     Secondary reactions = 0                                     ; 0/1 - use secondary reactions in calculations
         fiss_FilterUse0 = 1
         fiss_FilterUse1 = 1
         fiss_FilterUse2 = 1
            fiss_ellipse = 5
          fiss_NdeltaTop = 0
          fiss_ZdeltaTop = 0
          fiss_NdeltaBot = 25
          fiss_ZdeltaBot = 20
         frag_FilterUse0 = 3
         frag_FilterUse1 = 3
         frag_FilterUse2 = 3
            frag_ellipse = 4
          frag_NdeltaTop = 5
          frag_ZdeltaTop = 5
          frag_NdeltaBot = 6
          frag_ZdeltaBot = 6

[abrasion_fission]
... 2087 more lines ...
  104   Fri Dec 6 20:34:56 2019 CA, DK, NHDec 6 Night Shift
21.35 AIDA DAQ currently running, TapeService on, not writing to storage

      System wide checks ok

      Attachments 1-3: temperatures, good event stats, detector bias/leakage currents ok

22.04 waiting for beam. Possibility of fragments, further news to come

22.43 expecting fragments with 300 MeV/u 40Ar primary beam

      histograms zeroed

00.31 still waiting for beam

      detector bias/leakage currents ok (see google spreadsheet)

      FEE64 temps and good event statistics ok

02:02 still awaiting beam - issues with interlock

03:17 beam tuning - centering primary beam on target

03:43 beam reached AIDA, fragmentation beam (see elog entry by DK for PID)

      primary beam 300 Mev/u 40Ar, fragments in region of 34Si

      started writing to file Dec19/R2 at rate ~ 4000 kb/s

      histograms zeroed

03.49 system wide checks all ok

      attachments 4-6:

      temperatures ok

      bias/leakage currents ok

      good event statistics ok, except aida09 running slightly fast

04:03 attachments 7-8 - 1.8.H energy spectra

                      - no high energy events seen in aida10 -> checked ASIC control

                      - still no high energy events in aida10

04:09 attachment 9 - rate spectra for all FEE modules

04.16 attachment 10 - 1D hits for all FEE modules

04.19 beam stop - file R2_3

04:52 beam back on - file R2_6 - rate of ~ 5000 ions per spill

06:13 temperatures ok

      good event statistics ok (aida09 still running fast) - attachment 11

      detector bias/ leakage currents ok - see google spreadsheet

      all system wide checks ok

      1.8.H spectra - attachments 12/13

06:20 attachment 14 - rate spectra for all FEE modules

      attachment 15 - 1D hits for all FEE modules

      still no high energy events in aida10


      
Attachment 1: 191206_2139_temp.png
191206_2139_temp.png
Attachment 2: 191206_2140_bias.png
191206_2140_bias.png
Attachment 3: 191206_2140_stats.png
191206_2140_stats.png
Attachment 4: 191107_0351_temp.png
191107_0351_temp.png
Attachment 5: 191207_0352_bias.png
191207_0352_bias.png
Attachment 6: 191207_0352_stats.png
191207_0352_stats.png
Attachment 7: 191207_18h_2.png
191207_18h_2.png
Attachment 8: 191207_18H_1.png
191207_18H_1.png
Attachment 9: 191207_0408_rates.png
191207_0408_rates.png
Attachment 10: 191207_0415_statspec.png
191207_0415_statspec.png
Attachment 11: 191207_0614_stats.png
191207_0614_stats.png
Attachment 12: 191207_0606_18H_1.png
191207_0606_18H_1.png
Attachment 13: 191207_0609_18h_2.png
191207_0609_18h_2.png
Attachment 14: 191207_0618_rate.png
191207_0618_rate.png
Attachment 15: 191207_0619_hits.png
191207_0619_hits.png
  103   Thu Dec 5 22:00:11 2019 CA, DK, NHDec 5 Night Shift
15:30
  TD found the merger was running at ~43 Mb/s
  The odd-numbered FEE64s all had significant (>200k) disk rates
  ALL LEC/MEC fast discriminators and increased slow comparator from 0xa to 0x14
  Merger rate dropped to a more nominal value like 2.3 Mb/s
  Settings not saved

40Ar beam 300 MeV/u expected, but SIS problem at the moment.

23:00
  Leak currents are stable as previously (20-21 Nov) and entered to Google spreadsheet
  S4 vault is closed.
  Presently AIDA is running without tape storage
  System wide checks: 
    Master clock / aida01 sync reports error as expected; 
    All white rabbit decoders fail, but they are just off by 1 bit (no problem)
    Everything else okay.
  Stats look okay, but keep in mind above discriminator threshold -- see attachment 1
  Temperatures fine -- see attachment 2
  MBS looks okay -- we see a possible hint for the 1 bit difference with an "unexpected data word" -- see attachment 3 
  Data rate is 1.7 Mb/s, merger okay -- see attachment 4

23:30 
  Now we wait for the beam

00:05
  Attachments 5-11: results of system wide checks after collecting sync errors and WR errors from baseline - all ok

00:25
  Histograms zeroed at 00:25.

00:35

  1.8.L pulser peak widths

FEE64       width(ch)
  1            92
  2            131
  3            75
  4            97
  5            89
  6            117
  7            90
  8            87
  9            223
 10            58
 11            112
 12            126 

no peak observed on aida09

For comparison, pulser peaks measured on 13/11/19:

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

2:00
  40Ar beam tuning is finished.  Intensity at S4 is about 3 kHz per spill.  1 spill per / 10 seconds
  MBS system cannot be used for online merging tonight, because NeuLAND group left their connection open
  We can merge offline using the White Rabbit timestamps
  Remove the beam blocking steel flange from in front of AIDA et al.
  Writing to file Dec19/R1

03.20 Rate, Stat, 1.8.L, 1.8.H, 1.8.W spectra - attachments 13-20
      Observe
       light ions in 1.8.L spectra
       low energy (<1.5GeV) heavy ions in 1.8.H spectra with most heavy ions stopping in DSSSDs 1 & 2
       Stat hit patterns indicate broad (c. 5cm FWHM) centred heavy ion peak

03.24 Detector biases & leakage currents OK - attachment 21
      FEE64 temperatures OK - attachment 22
      Good event statistics (12 of 12 < 50k) - attachment 23

      System wide checks
      List All Sync Error Counters - attachment 24
      List all Sync Received Counters - attachment 25

      Baseline invoked earlier c. 00.05
      Collect all SYNC Error Counters for Baseline
      Collect All White Rabbit status Error Counters for Baseline 

04:07 2.6 mm Al degrader. 300Hz beam. 300 MeV/u 40Ar beam.

04.14 pulser peak seen in aida09 after performing ASIC check-load earlier in shift - width = 219 ch

04.45 1.8.H energy spectra (attachment 26/27)

06:08 attachments 28-30: detector bias/leakage currents ok
                         temperatures ok
                         good event statistics ok
                         all system wide checks ok

08:49 DAQ stop/start. TapeServer writing to No Storage. End of beam.


      
Attachment 1: 05-Dec-Stats.png
05-Dec-Stats.png
Attachment 2: 05-Dec-Temps.png
05-Dec-Temps.png
Attachment 3: 05-Dec-MBS.png
05-Dec-MBS.png
Attachment 4: 05-Dec-MergerRates.png
05-Dec-MergerRates.png
Attachment 5: adccalib.png
adccalib.png
Attachment 6: clockstatus.png
clockstatus.png
Attachment 7: meminfo.png
meminfo.png
Attachment 8: syncerrors.png
syncerrors.png
Attachment 9: syncpulses.png
syncpulses.png
Attachment 10: wrdecoder.png
wrdecoder.png
Attachment 11: 1.png
1.png
Attachment 12: 2.png
2.png
Attachment 13: 3.png
3.png
Attachment 14: 4.png
4.png
Attachment 15: 5.png
5.png
Attachment 16: 6.png
6.png
Attachment 17: 7.png
7.png
Attachment 18: 8.png
8.png
Attachment 19: 9.png
9.png
Attachment 20: 10.png
10.png
Attachment 21: 11.png
11.png
Attachment 22: 12.png
12.png
Attachment 23: 13.png
13.png
Attachment 24: 18hOdd.png
18hOdd.png
Attachment 25: even18h.png
even18h.png
Attachment 26: cja_bias.png
cja_bias.png
Attachment 27: cja_stats.png
cja_stats.png
Attachment 28: cja_temp.png
cja_temp.png
Attachment 29: pulser_peaks1.png
pulser_peaks1.png
Attachment 30: pulser_peaks2.png
pulser_peaks2.png
  102   Wed Nov 27 16:33:50 2019 NHNew Bias & Waveforms

HV core (-160 V) now attached to bottom 3 FEEs (with grounded kapton)

DSSDs bias OK, leakages unchanged

No appreciable change in waveforms

Some estimates

aida01/aida05/aida09 VHF noise: 5 channels frequency = 10 MHz ?

All FEEs:
Oscillation at ~25-35 channels = 1.4 to 2 MHz ?
Varies a bit with each FEE.

No obvious issue with upper FEEs unsure why 10 MHz is present - fee09 has been replaced but showed it before as well.

Lower frequency noise on every FEE may be ground noise from e.g. switching PSU?

No more access to S4 to check thoroughly again.

 

 

 

 

Attachment 1: waves_new_odd.png
waves_new_odd.png
Attachment 2: waves_new_even.png
waves_new_even.png
  101   Mon Nov 25 13:25:51 2019 NHWaveforms
> AIDA Waveforms immediately after beam at 21.11.19 8am or so.

Update 25.11.19 with all FEE64 waveforms added

Things to note:
aida01, aida05, aida09 are all on TOP - show VHF noise, esp. aida01?
TOP is where HV is connected (-ve core)

Everything noisy in general though
Attachment 1: wave_odd.png
wave_odd.png
Attachment 2: wave_even1.png
wave_even1.png
  100   Fri Nov 22 08:55:31 2019 NHWaveforms
AIDA Waveforms immediately after beam at 21.11.19 8am or so.
Attachment 1: waves_21119_even.png
waves_21119_even.png
Attachment 2: 21119_waves_odd.png
21119_waves_odd.png
Attachment 3: waves_21119_even_z.png
waves_21119_even_z.png
Attachment 4: waves_21119_odd_z.png
waves_21119_odd_z.png
  99   Wed Nov 20 20:36:14 2019 CA, CB, DK, NH20/21 November Night Shift
21:35 (NH) - A few merger timestamp errors were appearing - Decided to powercycle all FEE64s, all checks OK.
 No Screenshot due to forgetting before I closed merger 
 FEEs were often not stopping properly and needing merger reset as well.

Beam Plan:

40Ar 300 MeV/u Primary Beam
and/or
34Si Fragment from Ar

22:45

FEES were restart near these timestamps in the TapeData, evidently
-rw-rw-r--.  1 npg  npg  2.0G Nov 20 21:17 R1_361
-rw-rw-r--.  1 npg  npg  2.0G Nov 20 21:36 R1_362

Run number was kept the same (Tape Server was not stopped @ FEE restart?)

All system wide checks are okay (except of course the master clock fails)
Temps look okay (attachment 1, dated 21/Nov by mistake)
Leak currents all around 0.9 uA as before (attachment 2)
Stats are behaved. We don't have beam yet (attachment 3)
Merger and tape server happy (see attachment 4)
Data rate histos while beam is still off (see attachment 5)

==Nov 21==

0:00
TD noted that the data rates have gone up at some point (about double).  
Now we increased the slow comparator thresholds from 0xa to 0xb (data rate drops from 24 Mb/s to 16 Mb/s).
We increased it again from 0xb to 0xc, but near that time the beam was also turned on, and the data rate was about 25 Mb/s.

Beam was turned on near R1_472 or a bit earlier.

TD requested waveforms, but at the moment they are not appearing.  Will enable WFs after a condition change.

Beam seen in all three DSSSDs, see attachment 6.  Beam was turned off before I could get more screencaps.

Degrader change ... near R1_481

00:32 waveforms now enabled under Run Control.  They also appear to be enabled on ASIC1 undre LED and Waveform controls.  However, we are unable to see the waveforms in the relevant histograms.

00:44 3 g/cm^2 inserted R1_488

00:45 3.2 g/cm^2 degrader in R1_489.

1:00 Several degraders and combinations are being tested since last note.

1:10 It seems 1 g/cm^2 of energy loss is not accounted for by LISE++.  Trying to understand the reason.

1:28 Satisfied with a total degrader thickness of 2.6 g/cm^2 -- seems to calculate that fragments should implant mainly in DSSSD2?  Issue seems to be resolved updating the LISE++ file.
  Near R1_507

Can see the implant samples attached as #7 and #8

1:43 According to the stats, most of the beam energy is being deposited in DSSSD1.  See #9.
  Present beam burst repetition is around 0.1 Hz (each ~10 sec)

2:00 better example of stats histos (see #10)

2:45 tuning the FRS degraders (ones for RIB separation rather than AIDA implantation depth).

3:17 Just started fragments.  Disregard previous filenames with "34Si".  Finally no degrader in the FRS was needed to get Z separation.
  R1_559
  As the DAQs are not synchronized, a quick call to the BNC PB-5 pulser shifiting from 2 -> 50 -> 2 Hz was used to make a signal burst.
  S4 degrader is 5.2 g/cm^2

3:22 Updated biases in Google Sheets.

Having trouble seeing the fragments in the AIDA histograms.  Since we just started fragments, zero the histos.

3:35 Attachment 11, with 34Si fragments as Stats.  Difficult to click Update on the Rates at the right time.  
   From the beam monitors, fragments should be at around 3000 cps.

3:45 (Up to now, some optics tuning with slits, etc)

3:46 Optics settings accepted, AIDA at R1_572, accept this condition for data runs on ~34Si fragments.

3:47 Sent a pulser burst 2 -> 50 -> 2 Hz

3:50 zero histograms again.
   Temperatures attachment 12
   Biases attachment 13
   Statistics attachment 14
   Merger / Tape Server attachment 15

3:56   
   StatHistos attachment 16

4:09 FRS starts new run #56, AIDA R1_583.  Pulser 2-> 50 -> 2 Hz

4:56 Stat histograms in attachment 17.  Not zeroed since noted at 3:50
Attachment 1: 21-11-temps.png
21-11-temps.png
Attachment 2: 20-11-biases.png
20-11-biases.png
Attachment 3: 20-11-stats.png
20-11-stats.png
Attachment 4: 20-11-merger-ts.png
20-11-merger-ts.png
Attachment 5: 20-11-statshistos.png
20-11-statshistos.png
Attachment 6: 21-11-layout5histos.png
21-11-layout5histos.png
Attachment 7: 34si-implants-130-1.png
34si-implants-130-1.png
Attachment 8: 34si-implants-130-2.png
34si-implants-130-2.png
Attachment 9: 34si-stats-histos.png
34si-stats-histos.png
Attachment 10: stats-histos-2am.png
stats-histos-2am.png
Attachment 11: 34si-fragments-stats.png
34si-fragments-stats.png
Attachment 12: temperatures-fragments.png
temperatures-fragments.png
Attachment 13: biases-fragments.png
biases-fragments.png
Attachment 14: stats-fragments.png
stats-fragments.png
Attachment 15: merger-tapeserver-fragments.png
merger-tapeserver-fragments.png
Attachment 16: stathistos-fragments.png
stathistos-fragments.png
Attachment 17: stathistos-5am.png
stathistos-5am.png
  98   Tue Nov 19 21:22:06 2019 CA, CB, DK, NH19/20 November 2019 - overnight shift
19 November

Waiting for 40Ar beam to be delivered for tests.
Objective today is to assess degrader thickness suitable to stop 40Ar in 2nd DSSD layer.
Energy expected at 300 MeV/A, around 1E3-4 beam intensity - to be determined.

Tomorrow fragments could be expected.

DSSD / FEE64 pairing as follows

DSSD3 (most downstream): FEEs 9,10,11,12
DSSD2 (middle): FEEs 5,6,7,8
DSSD1 (most upstream): FEEs 1,2,3,4
+plastic scintillator furthest upstream

New AIDA Google spreadsheet created with leakage current, to monitor them during beamtime. First entry added. Bias attached.
https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023

DESPEC platform has already been moved. No effect on rates - see attached. FEE1 is missing one channel. FEE9, 10 noisy as expected. See attached.


System wide checks *all OK* except
Master clock failed (normal)
ADC calibration fails on 6 and 12. Recalibrated via FADC Align. All OK now.

MBS transfer OK. See attached.

Good event statistics OK. See attached.

FEE64 temperatures OK. See attached.

Merger and data rate OK. See attached.

Waiting for beam.



20 November

01:07 - Still no beam. 

System wide checks OK. 
Stats OK. 
Temps OK. 
MBS OK. 
Merger and data rate OK.


02:35  - Beam is expected soon
Tried enabling writing data to disk. Merger crashed. FEE64 stopped responding to commands.
Reset Merger. FEE64 now respond to commands.
Tried enabling writing to disk again. Merger crashed once more. Reset merger.
Reset Tape Server interface. Now writing to disk.

No data currently being sent to MBS, but operators say they "are adding a new detector". May be normal.

DAQ & Merger going OK. Writing to Nov19/R1

No beam yet.

Temperatures OK
System wide checks OK
Stats OK. See attached.
MBS not working.
Merger and data rate OK. See attached.

~02:48 Beam on. 40Ar. 300 MeV/A, 500 pps. Degrader settings unclear.

03:54 Degrader settings have been changing between 4.5, 1 and 6 ug/cm2
Currently on 6 g/cm2.
AIDA seen through MBS shows no HE on detector 3, and Det 1 and 2 show no counts on positive y (FEE6 and 1). That is not the case from MIDAS.

04:30 The aim of the tests above was to have no 40Ar at all in DSSD3. It does not appear that 6 g/cm2 is sufficient, as we still see events. The reduction between 1 & 2 vs. 3 is about a factor of 2-10 depending on ADC channel. See attached.

~04:30 Moved to 6.2 g/cm2 degrader. See attached.

~04:45 Moved to 5.8 g/cm2. See attached.

~05:09 Moved to 6.4 g/cm2. See attached.

05:31 High energy channel spectra (1.8.H) for all FEE64 w/ 6.4 g/cm2 degrader - See attachments 13 & 14

05:38: Moved to 1.0 g/cm2. See attached. Difference is not huge which may indicate these spectra are misleading, as partly expected. A more detailed analysis will be carried out off line.

06:00: Beam stopped. ~R77. AIDA keeps running background over the day. Beam should restart around 11 pm later today.
Attachment 1: Bias2235.png
Bias2235.png
Attachment 2: Rates2236.png
Rates2236.png
Attachment 3: 19112241MBS.png
19112241MBS.png
Attachment 4: 19112242Stats.png
19112242Stats.png
Attachment 5: 19112244Temperatures.png
19112244Temperatures.png
Attachment 6: 19112246Merger.png
19112246Merger.png
Attachment 7: 20110239.png
20110239.png
Attachment 8: 20110240DataTransfer.png
20110240DataTransfer.png
Attachment 9: 6gdegrader.png
6gdegrader.png
Attachment 10: 6.2degrader.png
6.2degrader.png
Attachment 11: 5.8degrader.png
5.8degrader.png
Attachment 12: 6.4degrader.png
6.4degrader.png
Attachment 13: 18h_0530.png
18h_0530.png
Attachment 14: 18H_0528.png
18H_0528.png
Attachment 15: 1gcm2degrader.png
1gcm2degrader.png
  97   Wed Nov 13 15:38:20 2019 NH[HowTo] MBS for AIDA (Nov 19)
For MBS integration AIDA forwards its data to an MBS Foreign Data Receiver (FDR), currently the PC assigned for this task is x86l-94.

To start the MBS system do the following:

Connect to the MBS FDR: ssh despec@x86l-94 (Ask local for password)

cd to the AIDA directory: cd mbsrun/nov19/aida_to_mbs

Start mbs: mbs

Startup the receiver: @startup

Now open the MBS relay for MIDAS (far right icon on the top)

Check both terminals (Relay & MBS) report a connection

In another terminal ssh into the FDR and run rate to monitor the data rate in MBS

For data to be transferred the Tape Server must be GOING but can be in No Storage mode if no MIDAS storage is required.

--

For experiments MBS file saving is handled by the DESPEC time sorter which merges all the subsystems together.
For testing you can write files from x86l-94 using the following commands

connect rfio XXX -disk
Where XXX is an Linux PC (ask local for a PC)

open file /path/to/file_ first=1 size=2000 -auto -rfio
This opens the file, writing in 2GB chunks

clo file
This will close the file when you are done

--

It is possible to restart the MBS relay at any point if the connection seems to have failed, also confirm the Tape Server is GOing
  96   Wed Nov 13 09:10:37 2019 CA, NH, OH13th November 2019
10:10 moved HV cables from 11 and 12 to 10 and 9 - all detectors biased in same configuration

      lowered slow comparator threshold to 0xa
   
      DAQ start

      pulser on

      zero'd histograms

pulser peak widths:
                   FEE      width (ch)

                   aida01 - 130

                   aida02 - 168
             
                   aida03 - 83

                   aida04 - 83

                   aida05 - 200

                   aida06 - 148

                   aida07 - 250

                   aida08 - 81

                   aida09 - 134

                   aida10 - 190

                   aida11 - 333

                   aida12 - 149

For comparison, peak widths from yesterday's measurement:

                   pulser peak widths -   FEE         width(ch)
                                           1             128
                                           2             145
                                           3             84
                                           4             74
                                           5             207
                                           6             125
                                           7             172
                                           8             80
                                           9             193
                                           10            183
                                           11            238
                                           12            138

Attachments 1 & 2 : pulser peak spectra

Attachment 3 - good event statistics

Attachment 4 - detector bias/ leakage currents

                   
Attachment 1: 191113_pulserpeaksOdd.png
191113_pulserpeaksOdd.png
Attachment 2: 191113_pulserpeaksEven.png
191113_pulserpeaksEven.png
Attachment 3: 191113_goodevents.png
191113_goodevents.png
Attachment 4: 191113_bias.png
191113_bias.png
  95   Wed Nov 13 01:54:00 2019 TDAida10 Unusual Pulser Peak

Double peaking normally indicates periodic baseline variations with period comparable to shaping time

Quote:

aida10 (no alpha rate recorded) has a strange pulser peak - looks like double peak?

Pulser terminator checked and OK...

All others "OK" but noise quite bad (see previous entry)

 

  94   Tue Nov 12 14:10:18 2019 NH, OH, CAAida10 Unusual Pulser Peak

aida10 (no alpha rate recorded) has a strange pulser peak - looks like double peak?

Pulser terminator checked and OK...

All others "OK" but noise quite bad (see previous entry)

Attachment 1: aida10-pulser-121119.png
aida10-pulser-121119.png
Attachment 2: pulsers121119.png
pulsers121119.png
Attachment 3: pulsersodd121119.png
pulsersodd121119.png
  93   Tue Nov 12 13:59:07 2019 NH, OH, CAAwfully noisy waveforms
Rates in all FEEs much higher than before - waveform shows very noisy in all systems. - attachment 1

Good event statistics - attachment 2

DISC in 7 and 3 above 300k

ASIC check load performed.

Good event statistics - attachment 3

Thesholds at 0xa for all FEE

Disc rate now 0 across all FEE


Pulser settings: attachment 4

pulser peak widths -   FEE         width(ch)
                              1             128
                              2             145
                              3             84
                              4             74
                              5             207
                              6             125
                              7             172
                              8             80
                              9             193
                              10            183
                              11            238
                              12            138

Attachments 5 & 6: pulser peaks for all FEE64

16.24: DAQ stop

       slow comparator threshold changed to 0x64

       ASIC check performed

       Alpha run/DAQ start (w/ data transfer enabled) - writing to file 31Oct19/R13

       Merger and TapeService ok - attachments 7 & 8
Attachment 1: awfulwaves121119.png
awfulwaves121119.png
Attachment 2: 191112_GoodEvents.png
191112_GoodEvents.png
Attachment 3: 191112_StatsPostLoad.png
191112_StatsPostLoad.png
Attachment 4: pulsersettings191112.png
pulsersettings191112.png
Attachment 5: pulserpeaksOdd.png
pulserpeaksOdd.png
Attachment 6: pulserpeaks_Even.png
pulserpeaks_Even.png
Attachment 7: merger191112.png
merger191112.png
Attachment 8: tapeservice191112.png
tapeservice191112.png
  92   Tue Nov 12 13:22:19 2019 OH, NH, CAAIDA09 Replacement
ASIC 1 on FEE9 was not producing signals.

A new HDMI cable was installed but still no signals.

AIDA09 was replaced with a new FEE.
Current list of FEE to serial numbers installed attachment 1

MAC address of new FEE was obtained.

Backup of DHCPD.conf was made dhcpd.confBACKUP191112

dhcpd.conf was updated with new MAC address.

All FEEs powered on and 09 was seen to mount and is seen by AIDAServer.

AIDA09 flashed to newest version of firmware. (0x18430701) - attachment 2
Attachment 1: FEE-SerialNumber.txt
FEE#	Messanie Serial No.	MAC Address
4	026-1690101-0016	d8-80-39-41-f6-b7
8	026-1690101-0013	d8-80-39-41-ba-2b
12	026-1690101-0030	d8-80-39-41-cf-ac
2	026-1690101-0011	d8-80-39-41-ba-22
6	026-1690101-0035	d8-80-39-41-ee-72
10	026-1690101-0032	d8-80-39-41-ee-10
1	026-1690101-0027	d8-80-39-41-ba-8a
5	026-1690101-0008	d8-80-39-41-d7-cc
9	026-1690101-0015	d8-80-39-41-f6-ee
7	Not checked	d8-80-39-41-b4-0c
11	026-1690101-0006	d8-80-39-41-f6-5a
3	026-1690101-0019	d8-80-39-41-d8-21
		
		
		
Dead FEE	Messanie Serial No.	MAC Address
ASIC 1 Dead	026-1690101-0005	d8-80-39-42-0d-0c
  91   Mon Nov 11 10:34:19 2019 NHAlpha Analysis (11.11.19)

Analysis of approx 20GB of data since Thursday night (not continuous - estimate of hours incominG)

Figures 1-3: Energy Front vs Energy Back (No F-B energy gate)

Figures 4-6: 2D hit pattern

Figures 7-9: 1D hit pattern

aida10 does not seem to be reading out still, unsure of reason? Looks fine with pulser.
Slow comparator not setting or unusual gain?

Otherwise good

Attachment 1: DSSD1-Energy.png
DSSD1-Energy.png
Attachment 2: DSSD2-Energy.png
DSSD2-Energy.png
Attachment 3: DSSD3-Energy.png
DSSD3-Energy.png
Attachment 4: DSSD1-2DHP.png
DSSD1-2DHP.png
Attachment 5: DSSD2-2DHP.png
DSSD2-2DHP.png
Attachment 6: DSSD3-2DHP.png
DSSD3-2DHP.png
Attachment 7: DSSD1-1DHP.png
DSSD1-1DHP.png
Attachment 8: DSSD2-1DHP.png
DSSD2-1DHP.png
Attachment 9: DSSD3-1DHP.png
DSSD3-1DHP.png
  90   Thu Nov 7 10:22:26 2019 Nic, PatrickReply: WR Timestamps
> > > All 12 FEEs have valid WR Timestamps
> > > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > > 
> > > HDMI cables in aida09 checked and good.
> > 
> > The problem would be the cable , one end or the other. 
> > I think ( if I recall ) a setup would restart the WR decoder.
> > 
> > I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
> 
> Cable will be replaced once a spare is available.
> 
> Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.
> 
> WR rate was chosen by Vic I believe, I am unsure of reasoning myself.

An update/clarification:
Although the upper bits are zero I believe actually it is a total failure to synchronise to WR:

aida09
WR Time Item 0x80500000 0x0fbd8000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80400249 0x0fbd8000; Time (28:47)=0x249; Time (0:27)=0x0fbd8000
WR Time Item 0x80500000 0x0fbdc000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80400249 0x0fbdc000; Time (28:47)=0x249; Time (0:27)=0x0fbdc000
WR Time Item 0x80500000 0x0fbe0000; Time (48:63)=0x0; Time (28:47)=0x249; Time (0:27)=0x0fbe0000
WR Time Item 0x80400249 0x0fbe0000; Time (28:47)=0x249; Time (0:27)=0x0fbe0000

aida10
WR Time Item 0x8050022e 0x0bde8000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x804e29d5 0x0bde8000; Time (28:47)=0xe29d5; Time (0:27)=0x0bde8000
WR Time Item 0x8050022e 0x0bdec000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x804e29d5 0x0bdec000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdec000
WR Time Item 0x8050022e 0x0bdf0000; Time (48:63)=0x22e; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000
WR Time Item 0x804e29d5 0x0bdf0000; Time (28:47)=0xe29d5; Time (0:27)=0x0bdf0000
  89   Tue Nov 5 11:15:41 2019 NH PJCSReport: aida10 database corruption

If ypou now check the actual values in the aida10 Options screen are the correct ?

It can happen that corruptions once loaded are propagated when Options are generally saved. 

Looking at the dates the files were modified does that make sense to you. It doesn't look as if the corrupted database was changed except when they all were ? Can you check ? Have you checked the other Options files ? I seem to recall that RIKEN AIDA had a program to to this ?

Quote:

Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.

The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.

 

Quote:

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could welll cause you problems with readout as it won't understand some of the new data.

 

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

 

 

  88   Tue Nov 5 10:32:21 2019 NH PJCSReport: aida10 database corruption

Hi, see the attached files in the original comment. I think the BACKUPCorrupt is quite old now.

The Options data gets corrupted and for example the ASIC folder gets set to 'undefined' which I think is why the ASIC data loads incorrectly.

 

Quote:

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could welll cause you problems with readout as it won't understand some of the new data.

 

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

 

  87   Tue Nov 5 10:02:45 2019 NH PJCSReport: aida10 database corruption

This looks like the Options not the ASIC data. That will be in the CONTENTS file of the FEE64 named directory within the dated EXPERIMENT directory. Looks like the latest is at /MIDAS/DB/EXPERIMENTS/AIDA/2019Oct31-13.24.23/aida10 ?

A quick look at this file and it looks normal , not all 0xad ? 

Looked at the Options file in the /MIDAS/DB/EXPERIMENTS/AIDA/Options.BACKUPCorrupt/aida10 and the worst bit is the name of the Data Acquistion program.  That could well cause you problems with readout as it won't understand some of the new data.

Hope this helps.

Quote:

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

 

  86   Mon Nov 4 13:17:12 2019 NHReport: aida10 database corruption

Often aida10 database seems to get corrupted (ASIC values are all 0xad)
Unsure why it's only aida10, might be related to weird alpha rate behaviour at the moment.

 

Attached is corrupted database and correct database

Attachment 1: db_working.txt
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&RunNumber&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida.Vchannels&&ASIC.settings&&Aida_Hist_D_Enable&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&DataAcqPgm&&Aida_Hist_H_Enable&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 1
Aida.shift string 0
TS_SYNC_PHASE string 0x0
ExtClk string 1
0x0006dead string 0x0000
Aida.offset string 0
MACB_TRIG_MODE string 4
Aida_GroupBase string 1
Stat.channels string 512
RunNumber string 12
Rate.channels string 512
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
ASIC.settings string 2019Oct31-13.24.23
Aida_Hist_D_Enable string 1
Stat.shift string 6
Aida.Wchannels string 1020
Aida.channels string 65536
Aida_Hist_V_Enable string 0
Include.Aida string 0
Aida_Hist_H_Enable string 1
DataAcqPgm string AidaExecV9
DataFormat string 0x0000
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
Attachment 2: db_corrupt.txt
Index string Stat.offset&&Aida_Hist_W_Enable&&TS_SYNC_PHASE&&ExtClk&&Aida.shift&&MACB_TRIG_MODE&&Aida.offset&&0x0006dead&&Aida_GroupBase&&Rate.channels&&Stat.channels&&Aida_Hist_D_Enable&&Aida.Vchannels&&ASIC.settings&&WAVE_DMA_HWM&&Stat.shift&&Aida.Wchannels&&Aida_Hist_V_Enable&&Include.Aida&&Aida.channels&&Aida_Hist_H_Enable&&DataAcqPgm&&DataFormat&&ASIC_DMA_HWM&&Aida_Hist_L_Enable
Stat.offset string 64
Aida_Hist_W_Enable string 0
Aida.shift string 1
TS_SYNC_PHASE string 1
ExtClk string 1
Aida.offset string 0
MACB_TRIG_MODE string 14
0x0006dead string 0x0000
Stat.channels string 150
Aida_GroupBase string 1
Rate.channels string 150
RunNumber string 2
Aida_Hist_D_Enable string 1
ASIC.settings string undefined
WAVE_DMA_HWM string 0x0007ffff
Aida.Vchannels string 256
Stat.shift string 6
Aida.Wchannels string 1020
Include.Aida string 1
Aida.channels string 32768
Aida_Hist_V_Enable string 0
Aida_Hist_H_Enable string 0
DataFormat string AIDA
DataAcqPgm string AidaExecV9
ASIC_DMA_HWM string 0x000fffff
Aida_Hist_L_Enable string 1
  85   Mon Nov 4 08:48:18 2019 NHAlpha Status 04.11.2019

Alpha run has been running over the weekend. Stats looking positive with notable exception that aida10 is not sending much data. Will attempt a powercycle.

Temperatures and Leakage Currents very good to excellent

Noted that the slow rate of aida10 seems to be slowing down the rate data is sent to tape - perhaps being buffered until an ADC event arrives?

Update 05.11.2019

Most rates OK, aida10 only recorded 1 event over night, aida09 spectra had a number of weird channels.
Performed ASIC check/load, aida10 found another 201 events but mostly in the HEC channel (as did all FEEs)
All other FEEs look very nice however

 

15:44 CET: Check/load performed as aida07 rate went very high (in HEC it seems) - now back to 0

 

Update 07.11.2019
Run stopped to move DESPEC platform, overnight the USB relay disconnected/reconnected powering down the FEEs (probably someone knocked the interlock wire)

 

Attachment 1: alphas-bias.png
alphas-bias.png
Attachment 2: alphas-temp.png
alphas-temp.png
Attachment 3: alpha-aida10.png
alpha-aida10.png
Attachment 4: stats-0511.png
stats-0511.png
Attachment 5: spec-0511.png
spec-0511.png
  84   Fri Nov 1 18:09:03 2019 CA, TD, NHSummary - 29.10-1.11.19
> >
> > 
> > - evidence of c. 1MHz extrinsic noise for most FEE64s
> 
> 
> Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.

Judge for yourself

https://elog.ph.ed.ac.uk/DESPEC/191031_125102/1350_18W.png

I would estimate c. 2.5 cycles in 2us => c. 1.2MHz … ?

Note that all of the waveforms are shown on an expanded scale c. 7000-9000 of 0-16383
so the amplitude is significantly less than that observed at LYCCA.

Tom
  83   Fri Nov 1 15:45:04 2019 CA, TD, NHSummary - 29.10-1.11.19
>
> 
> - evidence of c. 1MHz extrinsic noise for most FEE64s


Is the 1MHz noise actually 1.58Mhz? as this is the frequency of observed noise at LYCCA.
  82   Fri Nov 1 15:24:40 2019 CA, TD, NHSummary - 29.10-1.11.19
3x MSL type BB18(DS)-1000 installed 

detector bias -160V, leakage current c. 1uA @ +21 deg C for all 3x DSSSDs

all FEE64 good event rates (slow comparator 0xa, LEC fast comparator 0xff) c. 120k, or less, typically 30-50k, overall merge rate c. 1.6M data item/s

See https://elog.ph.ed.ac.uk/DESPEC/70



Outstanding issues

- occasional loss of bits 48-63 of WR timestamp for aida09 - can be recovered by power cycle - replace HDMI cable?

- aida09 asic 1 not producing ADC or disc data - replace aida09? 
  ASIC Check works OK
  aida09 asic temp c. 30deg C < other asic temps

- evidence of c. 1MHz extrinsic noise for most FEE64s

- acquisition of waveform data not robust - most ASIC channels do not produce data or quickly stop producing data  - PJCS to review firmware rev for Jan/Feb 2020

- require spare DSSSDs, FEE64s, HDMI cables etc
  81   Fri Nov 1 14:17:15 2019 Nic, PatrickReply: WR Timestamps
> > All 12 FEEs have valid WR Timestamps
> > Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> > 
> > HDMI cables in aida09 checked and good.
> 
> The problem would be the cable , one end or the other. 
> I think ( if I recall ) a setup would restart the WR decoder.
> 
> I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

Cable will be replaced once a spare is available.

Setup did not restart the WR decoder when this problem occurred beforehand - Reset/Setup tried.

WR rate was chosen by Vic I believe, I am unsure of reasoning myself.
  80   Fri Nov 1 10:46:13 2019 CA, TD, NHFriday 1st November 2019
11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
        before removing aluminium foil upstream of AIDA

        note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF

11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
        after removing aluminium foil upstream of AIDA

      - base of AIDA snout has Mylar shielding
Attachment 1: biaswfoil.png
biaswfoil.png
Attachment 2: statswfoil.png
statswfoil.png
Attachment 3: tempwfoil.png
tempwfoil.png
Attachment 4: biaswofoil.png
biaswofoil.png
Attachment 5: statswofoil.png
statswofoil.png
Attachment 6: tempswofoil.png
tempswofoil.png
  79   Fri Nov 1 09:30:17 2019 NHDatabase error stopping run?
Strange error about Database options appearing during run stop.
Does not seem to affect stopping/starting/data?
Attachment 1: ErrorStopDatabase.png
ErrorStopDatabase.png
  78   Thu Oct 31 19:02:42 2019 PatrickWR Timestamps
> All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> 
> HDMI cables in aida09 checked and good.

The problem would be the cable , one end or the other. 
I think ( if I recall ) a setup would restart the WR decoder.

I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?
  77   Thu Oct 31 17:13:54 2019 NHAlpha Run
Beginning Alpha run

Thresholds set to 100 (LEC/Slow) 255 (LEC/Fast) 2 (HEC)

Pulser is OFF

Recording to MIDAS: /TapeData/31Oct19/R1

Recording to MBS: /lustre/gamma/nhubbard/AIDA/Alphas311019_

Disk rate approximately 0 as expected.

01.11.19 19:11 CET: FEE10 Stat histogram was empty - performed a check/load and now rare events come in - Merger also emitted 10 blocks to storage quickly (presumably waiting on FEE10 data)
All FEEs now slowly showing alpha data
  76   Thu Oct 31 16:43:03 2019 NH CA TDMerger & MBS Performance
Testing merger with 3 DSSDs connected, with thresholds of 10 (slow comparator), 2 (HEC) and 255 (fast discriminator)

Merger handling rate of 1.6 million events per second comfortably. Forwarding to MBS fine.

Network usage: 130 Mbps (AIDA network) and 20 Mbps (MBS network)
Attachment 1: MergerStats.png
MergerStats.png
Attachment 2: MBSRates.png
MBSRates.png
Attachment 3: Mergernetworking.png
Mergernetworking.png
Attachment 4: MergerStatistics.png
MergerStatistics.png
  75   Thu Oct 31 15:24:35 2019 TDwaveform spectra issues?
Attachment 1: 20.png
20.png
Attachment 2: 21.png
21.png
Attachment 3: 22.png
22.png
Attachment 4: 23.png
23.png
Attachment 5: 30.png
30.png
Attachment 6: 31.png
31.png
Attachment 7: 32.png
32.png
Attachment 8: 33.png
33.png
Attachment 9: 40.png
40.png
Attachment 10: 41.png
41.png
Attachment 11: 42.png
42.png
Attachment 12: 43.png
43.png
  74   Thu Oct 31 14:27:06 2019 NHWR Timestamps
All 12 FEEs have valid WR Timestamps
Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.

HDMI cables in aida09 checked and good.
Attachment 1: WRTimes.png
WRTimes.png
  73   Thu Oct 31 14:23:37 2019 TDaida09 asic 1 no data readout?
Attachment 1: 10.png
10.png
Attachment 2: 11.png
11.png
Attachment 3: 12.png
12.png
Attachment 4: 13.png
13.png
  72   Thu Oct 31 10:33:34 2019 TD[How To] Start TclHttpd server on Raspberry Pi
1) Login to Raspberry Pi 

ssh pi@nnrpi2

2) Kill any instances of the TclHttpd server

ps -ef | grep TclHttpd
kill -9 <pid>

3) Start server

/MIDAS/TclHttpd/linux-arm/TclHttpd-server &
  71   Thu Oct 31 09:36:37 2019 TDRIKEN LayOut directory
Attachment 1: LayOut.tar.gz
  70   Thu Oct 31 09:00:58 2019 CA, TD, NHThursday 31 October
10.05 DSSSD # 1-3 detector biases & leakage currents OK - see attachment 1
      Ambient temperature +21.2 deg C, d.p. +3.4 deg C, RH 31.2%

10.15 DSSSD stack as follows

      DSSSD  serial  thickness  depletion 
      #              (um)       voltage (V)
      1      3208-7  1019       125              upstream
      2      3208-13 1020       120
      3      3208-14 1020       130              downstream

11.30 AIDA power relays sequenced on
      
      DAQ start

      system wide checks ok

11.33 attachment 2 - fee temperatures - ok

      attachment 3 - good event statistics with slow comparator threshold = 100

      attachment 4 - good event statistics with slow comparator threshold = 50

      attachment 5 - good event statistics with slow comparator threshold = 20

      attachment 6 - good event statistics with slow comparator threshold = 10
     
13.51 attachment 7 - Rates
                   - aida09 1 ASIC not working (no data)

      attachment 8 - good event statistics

      attachments 9/10 - 1.8.L spectra
                       - no peak observed in aida09

      attachments 11/12 - 1.8.W spectra
                        - waveform only observed in naida 2,4,7,8,12

      pulser peak widths -   FEE         width(ch)
                              1             71
                              2             96
                              3             66
                              4             79
                              5             74
                              6             96
                              7             89
                              8             83
                              9           no peak
                              10            82
                              11            153
                              12            120

17.26 FATIMA moved upstream around AIDA

      attachment 13 - good event statistics

      ASIC check-load performed

      attachment 14 - good event statistics following checkload
Attachment 1: 1.png
1.png
Attachment 2: 31OctTemp1119.png
31OctTemp1119.png
Attachment 3: stats31Oct1128.png
stats31Oct1128.png
Attachment 4: 50stats.png
50stats.png
Attachment 5: 20stats.png
20stats.png
Attachment 6: 10stats.png
10stats.png
Attachment 7: 1335rates.png
1335rates.png
Attachment 8: 1337_stats.png
1337_stats.png
Attachment 9: 1341_pulser.png
1341_pulser.png
Attachment 10: 1345pulser.png
1345pulser.png
Attachment 11: 1348_18W.png
1348_18W.png
Attachment 12: 1350_18W.png
1350_18W.png
Attachment 13: 1728_stats.png
1728_stats.png
Attachment 14: 1730_stats.png
1730_stats.png
  69   Wed Oct 30 11:51:08 2019 CA, TD, NHDetector biasing
12.51 - detector bias and leakage currents - detector 3 has no current
                                           - other detectors ok

13.13 - tried changing from HV3 cables to HV4 - still open circuit 

      - tried different permutations of moving HV-4 braid and core cables (high voltage to ground connections)

       (dssd3)

      (all directions defined as if looking upstream)

      - core -> bottom, braid -> left     open - circuit

      - core -> top, braid -> right       trips at 8V

      - core -> bottom, braid -> right     open - circuit

      - core -> top, braid-> left          trips at 8V

      - removed jumpers - still trips

13.36 - testing continuity of bond wires on dsssd 3208-14 & 3208-15

      - observe 20 ohm resistance on p+p and p+n bond wires 

19:41 - dsssd 1 and 2 not biasing, 3 ok.

      - changed bias configuration -> HV core 1 & 2 now on top, braid on right

      - ribbon cables for DSSSD #1 & #2 p+n junction strips down were misaligned

      - all detectors biased succesfully

HV cable configuration (looking upstream): HV-1 Core -> top-inner (FEE 1)
                        
                                           HV-2 Core -> top-middle (FEE 5)

                                           HV-3 Core -> bottom-outer (FEE 11)

                                           HV-1 Braid -> right-inner (FEE 2)

                                           HV-2 Braid -> right-middle (FEE 6)

                                           HV-3 Braid -> left-outer (FEE 12)

Attachment 3 - rough sketch of HV cable configuration, looking upstream.
       
Attachment 1: bias30Oct.png
bias30Oct.png
Attachment 2: 30Octbias2.png
30Octbias2.png
Attachment 3: hvsketch.pdf
hvsketch.pdf
  68   Wed Oct 30 08:06:49 2019 CA, TD, NHDSSSD stack
Ribbon cables fitted with kapton PCB couplers -> 3 isolated + 1 non-isolated for each cable length/DSSSD

Ribbon cable lengths:

without kapton PCB coupler (cm) - 40, 47, 51.5

with kapton PCB coupler (cm) - 50.5, 57.5, 62

gap between plastic and dsssd - 15 mm

length between top of snout and dsssd - 5cm
Attachment 1: IMG_20191030_091729.jpg
IMG_20191030_091729.jpg
Attachment 2: IMG_20191030_091734.jpg
IMG_20191030_091734.jpg
Attachment 3: IMG_20191030_091925.jpg
IMG_20191030_091925.jpg
Attachment 4: IMG_20191030_091930.jpg
IMG_20191030_091930.jpg
Attachment 5: IMG_20191030_102326.jpg
IMG_20191030_102326.jpg
Attachment 6: IMG_20191030_102329.jpg
IMG_20191030_102329.jpg
  67   Mon Oct 28 17:02:11 2019 TDAIDA @ DESPEC cable length estimates
AIDA snout length 37cm (from end of AIDA snout support assembly)

downstream scintillator (assembly depth 1cm)   36cm

downstream dsssd (#1)                          35.2cm

                 (#2)                          34.3cm

                 (#3)                          33.4cm

upstream dsssd   (#4)                          32.5cm

upstream scintillator (assembly depth 1cm)     31.7cm

Cable lengths required are therefore

downstream dsssd (#1)           35.2 - 10.5 + 2.15 + 5 + 22.5 = 54.35cm 21.4"

                 (#2)           34.3 - 10.5 + 2.15 + 5 + 16.5 = 47.45cm 18.7"

                 (#3)           33.4 - 10.5 + 2.15 + 5 + 10.5 = 40.55cm 16.0"

upstream dsssd   (#4)           32.5 - 10.5 + 2.15 + 5 + 4.5 = 33.65cm 13.2"

length of RH Kapton PCB coupler = 10.5cm (Carlo can you check please?)

turning the corner at the base of the AIDA snout support assembly 2.15+5=7.15cm
(see Peter's drawing)

distances to FEE64s 4.5, 10.5, 16.5 and 22.5cm
(see Peter's drawing)

f we use 3x DSSSDs then the cable lengths required will be as follows:

downstream dsssd (#1)           35.2 - 10.5 + 2.15 + 5 + 16.5 = 48.35cm 19.0"

                 (#2)           34.3 - 10.5 + 2.15 + 5 + 10.5 = 41.45cm 16.3"

upstream dsssd   (#3)           33.4 - 10.5 + 2.15 + 5 + 4.5 = 34.55cm 13.6"
  66   Mon Sep 16 12:24:46 2019 NHAIDA Monitoring

https://docs.google.com/spreadsheets/d/1fZBybHodSvacUdKpDM_VKznqTrh49CKnqKVQVMEDWGs/edit?usp=sharing

A spreadsheet of ambient conditions and water temperature in S4 for AIDA.
Will show the distance to danger points (interlock shutdown and condensation on the pipes)

ELOG V3.1.4-unknown