AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC, Page 25 of 37  ELOG logo
ID Date Author Subjectdown
  119   Fri Dec 20 12:04:03 2019 NH, TDFriday 20 December 2019
Fig 1: DSSD Leakage currents ok

Fig 2: FEE Temps ok

Fig 3: aida08 has a few timestamp errors in merger

Fig 4, 5: GSI WR OK

Fig 6: All System wide checks passed 

FIg 7: Stats good

13:11 - Alpha Run Stopped, Final File: R9_6

FEEs powered off, DSSD bias off.
Attachment 1: Screenshot-CAEN_HV_Control.png
Screenshot-CAEN_HV_Control.png
Attachment 2: FeeTemps.png
FeeTemps.png
Attachment 3: mergerTSerrors.png
mergerTSerrors.png
Attachment 4: gsiwr1.png
gsiwr1.png
Attachment 5: gsiwr2.png
gsiwr2.png
Attachment 6: checks.png
checks.png
Attachment 7: stats.png
stats.png
  80   Fri Nov 1 10:46:13 2019 CA, TD, NHFriday 1st November 2019
11.46 - attachments 1,2,3: bias/leakage currents, good event statistics and fee temperatures (respectively)
        before removing aluminium foil upstream of AIDA

        note - slow comparator threshold 0x64 (alpha background), LEC fast comparator threshold 0xff, pulser OFF

11.52 - attachments 4,5,6: bias/leakage currents, good event statistics and fee temperatures (respectively)
        after removing aluminium foil upstream of AIDA

      - base of AIDA snout has Mylar shielding
Attachment 1: biaswfoil.png
biaswfoil.png
Attachment 2: statswfoil.png
statswfoil.png
Attachment 3: tempwfoil.png
tempwfoil.png
Attachment 4: biaswofoil.png
biaswofoil.png
Attachment 5: statswofoil.png
statswofoil.png
Attachment 6: tempswofoil.png
tempswofoil.png
  205   Fri Mar 19 17:16:45 2021 TDFriday 19 March
Screenshots of successive displays of Options for FEE64s aida01, 02, 03 and 04.
Attachment 1: Screenshot_2021-03-19_Options_aidas-gsi.png
Screenshot_2021-03-19_Options_aidas-gsi.png
Attachment 2: Screenshot_2021-03-19_Options_aidas-gsi(4).png
Screenshot_2021-03-19_Options_aidas-gsi(4).png
Attachment 3: Screenshot_2021-03-19_Options_aidas-gsi(1).png
Screenshot_2021-03-19_Options_aidas-gsi(1).png
Attachment 4: Screenshot_2021-03-19_Options_aidas-gsi(5).png
Screenshot_2021-03-19_Options_aidas-gsi(5).png
Attachment 5: Screenshot_2021-03-19_Options_aidas-gsi(2).png
Screenshot_2021-03-19_Options_aidas-gsi(2).png
Attachment 6: Screenshot_2021-03-19_Options_aidas-gsi(6).png
Screenshot_2021-03-19_Options_aidas-gsi(6).png
Attachment 7: Screenshot_2021-03-19_Options_aidas-gsi(3).png
Screenshot_2021-03-19_Options_aidas-gsi(3).png
Attachment 8: Screenshot_2021-03-19_Options_aidas-gsi(7).png
Screenshot_2021-03-19_Options_aidas-gsi(7).png
  580   Fri Apr 19 13:52:34 2024 JB, CC, TDFriday 19 April contd.
12.00 Replaced AIDA ASIC mezzanine of aida01 to fix issue with asic #3

      During replacement the HDMI connector of the adjacent FEE64 aida09 became disconnected from the FEE64 PCB
      aida09 ( MAC ee:10 ) replaced (MAC 41:cf:ad )

      AIDA FEE64 adaptor PCBs for aida01, aida14 and aida09 disconnected and re-connected during this process 

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temps OK - attachment 2
       *except* aida02 ASIC temp - known fault

      System wide checks OK *except* WR/FPGA errors - attachments 3-4

      WR timestamps OK - attachment 5

      ADC, DISC, PAUSE, RESUME and correlation data stats - attachments 6-10
       ADC data items 10/16 < 20k, max 143k

      per FEE64 Rate spectra - attachment 11

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-13


15.10 Synchronise ASIC clocks
      Re-calibrate ALL ADCs 

      ASIC settings 2024Mar27-11.25.32

      Changes ( https://elog.ph.ed.ac.uk/CARME/499 )

       IBias LF feedback from 0xf to 0x8
       Diode link threshold from 0xbf to 0xca

      New ASIC settings saved as 2024Apr19-15.22.49

      All FEE64 slow comparators -> 0x14

      Data file S100_alpha/R12

      Pulser ealkthrough (test +)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity +
      Tail pulse
      tau_d 1ms

16.35 DAQ ends OK file S100_alpha/R12_24

      per p+n FEE64 1.8.L spectra - attachment 14
       aida09 pulser peak width 54 ch FWHM 

17.38 Data file S100_alpha/R13

      Pulser walkthrough (test -)

      BNC PB-5 settings
 
      Amplitude 10.0-1.0V @ 1.0V step
      Attenuation x10
      Frequency 25Hz
      Polarity -
      Tail pulse
      tau_d 1ms

18.43 DAQ ends OK file S100_alpha/R13_20

      per p+n FEE64 1.8.L spectra - attachment 15

18.55 Current status

      FEE64 power ON
      DSSSD bias ON

      All FEE64 slow comparator 0x14

      DAQ going -> Merger -> TapeServer (no storage mode) -> MBS ( but data stream not yet being read by MBS )


To Do

- S4 currently closed/controlled access for FRS startup

If we have access to S4 tomorrow

- switch test - to test + ( not critical )
- test AIDA interlock 
- further alpha background?



22.47 DSSSD bias & leakage current OK - attachments 16-17

      FEE64 temps OK - attachment 18
       *except* aida02 ASIC temp - known fault

      ADC data item stats - attachment 19

      per FEE64 Rate spectra - attachment 20

      Data link, Tape Server & Merger - attachments 21-23

       
Attachment 1: Screenshot_from_2024-04-19_14-56-51.png
Screenshot_from_2024-04-19_14-56-51.png
Attachment 2: Screenshot_from_2024-04-19_14-57-04.png
Screenshot_from_2024-04-19_14-57-04.png
Attachment 3: Screenshot_from_2024-04-19_14-58-27.png
Screenshot_from_2024-04-19_14-58-27.png
Attachment 4: Screenshot_from_2024-04-19_14-58-23.png
Screenshot_from_2024-04-19_14-58-23.png
Attachment 5: Screenshot_from_2024-04-19_14-59-02.png
Screenshot_from_2024-04-19_14-59-02.png
Attachment 6: Screenshot_from_2024-04-19_15-01-31.png
Screenshot_from_2024-04-19_15-01-31.png
Attachment 7: Screenshot_from_2024-04-19_15-01-17.png
Screenshot_from_2024-04-19_15-01-17.png
Attachment 8: Screenshot_from_2024-04-19_15-01-02.png
Screenshot_from_2024-04-19_15-01-02.png
Attachment 9: Screenshot_from_2024-04-19_15-00-39.png
Screenshot_from_2024-04-19_15-00-39.png
Attachment 10: Screenshot_from_2024-04-19_14-59-51.png
Screenshot_from_2024-04-19_14-59-51.png
Attachment 11: Screenshot_from_2024-04-19_15-02-09.png
Screenshot_from_2024-04-19_15-02-09.png
Attachment 12: Screenshot_from_2024-04-19_15-04-36.png
Screenshot_from_2024-04-19_15-04-36.png
Attachment 13: Screenshot_from_2024-04-19_15-03-48.png
Screenshot_from_2024-04-19_15-03-48.png
Attachment 14: Screenshot_from_2024-04-19_16-56-36.png
Screenshot_from_2024-04-19_16-56-36.png
Attachment 15: Screenshot_from_2024-04-19_18-45-05.png
Screenshot_from_2024-04-19_18-45-05.png
Attachment 16: Screenshot_from_2024-04-19_22-47-43.png
Screenshot_from_2024-04-19_22-47-43.png
Attachment 17: Screenshot_from_2024-04-19_22-48-52.png
Screenshot_from_2024-04-19_22-48-52.png
Attachment 18: Screenshot_from_2024-04-19_22-49-31.png
Screenshot_from_2024-04-19_22-49-31.png
Attachment 19: Screenshot_from_2024-04-19_22-50-15.png
Screenshot_from_2024-04-19_22-50-15.png
Attachment 20: Screenshot_from_2024-04-19_22-51-23.png
Screenshot_from_2024-04-19_22-51-23.png
Attachment 21: Screenshot_from_2024-04-19_22-52-39.png
Screenshot_from_2024-04-19_22-52-39.png
Attachment 22: Screenshot_from_2024-04-19_22-52-14.png
Screenshot_from_2024-04-19_22-52-14.png
Attachment 23: Screenshot_from_2024-04-19_22-51-55.png
Screenshot_from_2024-04-19_22-51-55.png
  579   Fri Apr 19 02:35:14 2024 TDFriday 19 April
03.30


09.10



09.25 data file S100_alpha/R11
   
      BNC PB-5 pulser data

      amplitude 10.0V
      attenuation x10
      tau_d = 1ms
      polarity - ( initially + )
      frequency 25Hz

      ASIC settings2024Mar27-11.25.32
       slow comparator 0x64 all FEE64s


      aida02 pulser peak width 129 ch FWHM

      polsrity +

      aida09 pulser peak width 57 ch FWHM


09.40 switch to no storage

      p+n FEE64 slow comparators 0x64 -> 0xa
      n+n FEE64 slow comparators 0x64 -> 0xf

      radioactive source 152Eu 370kBq installed on the bottom side of the snout, centred tranversely, located beneath bplas-DSSSD-DSSSD-bplas position.

      ADC data item stats - attachment 35
      9/16 < 20k, max 108k

      per FEE64 Rate spectra - attachment 36

10.44 152Eu source removed

      ADC data item stats - attachment 37
      9/16 < 20k, max 102k

      per FEE64 Rate spectra - attachment 38
Attachment 1: Screenshot_from_2024-04-19_03-31-31.png
Screenshot_from_2024-04-19_03-31-31.png
Attachment 2: Screenshot_from_2024-04-19_03-30-58.png
Screenshot_from_2024-04-19_03-30-58.png
Attachment 3: Screenshot_from_2024-04-19_03-30-45.png
Screenshot_from_2024-04-19_03-30-45.png
Attachment 4: Screenshot_from_2024-04-19_03-27-45.png
Screenshot_from_2024-04-19_03-27-45.png
Attachment 5: Screenshot_from_2024-04-19_03-26-26.png
Screenshot_from_2024-04-19_03-26-26.png
Attachment 6: Screenshot_from_2024-04-19_03-25-53.png
Screenshot_from_2024-04-19_03-25-53.png
Attachment 7: Screenshot_from_2024-04-19_03-24-58.png
Screenshot_from_2024-04-19_03-24-58.png
Attachment 8: Screenshot_from_2024-04-19_09-08-46.png
Screenshot_from_2024-04-19_09-08-46.png
Attachment 9: Screenshot_from_2024-04-19_09-09-19.png
Screenshot_from_2024-04-19_09-09-19.png
Attachment 10: Screenshot_from_2024-04-19_09-10-19.png
Screenshot_from_2024-04-19_09-10-19.png
Attachment 11: Screenshot_from_2024-04-19_09-10-10.png
Screenshot_from_2024-04-19_09-10-10.png
Attachment 12: Screenshot_from_2024-04-19_09-10-03.png
Screenshot_from_2024-04-19_09-10-03.png
Attachment 13: Screenshot_from_2024-04-19_09-10-51.png
Screenshot_from_2024-04-19_09-10-51.png
Attachment 14: Screenshot_from_2024-04-19_09-11-08.png
Screenshot_from_2024-04-19_09-11-08.png
Attachment 15: Screenshot_from_2024-04-19_09-10-59.png
Screenshot_from_2024-04-19_09-10-59.png
Attachment 16: Screenshot_from_2024-04-19_09-11-20.png
Screenshot_from_2024-04-19_09-11-20.png
Attachment 17: Screenshot_from_2024-04-19_09-11-46.png
Screenshot_from_2024-04-19_09-11-46.png
Attachment 18: Screenshot_from_2024-04-19_09-11-31.png
Screenshot_from_2024-04-19_09-11-31.png
Attachment 19: Screenshot_from_2024-04-19_09-17-48.png
Screenshot_from_2024-04-19_09-17-48.png
Attachment 20: Screenshot_from_2024-04-19_09-17-22.png
Screenshot_from_2024-04-19_09-17-22.png
Attachment 21: Screenshot_from_2024-04-19_09-14-14.png
Screenshot_from_2024-04-19_09-14-14.png
Attachment 22: Screenshot_from_2024-04-19_09-13-23.png
Screenshot_from_2024-04-19_09-13-23.png
Attachment 23: Screenshot_from_2024-04-19_09-12-08.png
Screenshot_from_2024-04-19_09-12-08.png
Attachment 24: Screenshot_from_2024-04-19_09-12-01.png
Screenshot_from_2024-04-19_09-12-01.png
Attachment 25: Screenshot_from_2024-04-19_09-11-54.png
Screenshot_from_2024-04-19_09-11-54.png
Attachment 26: Screenshot_from_2024-04-19_09-18-40.png
Screenshot_from_2024-04-19_09-18-40.png
Attachment 27: Screenshot_from_2024-04-19_09-18-22.png
Screenshot_from_2024-04-19_09-18-22.png
Attachment 28: Screenshot_from_2024-04-19_09-18-01.png
Screenshot_from_2024-04-19_09-18-01.png
Attachment 29: Screenshot_from_2024-04-19_09-29-33.png
Screenshot_from_2024-04-19_09-29-33.png
Attachment 30: Screenshot_from_2024-04-19_09-27-40.png
Screenshot_from_2024-04-19_09-27-40.png
Attachment 31: Screenshot_from_2024-04-19_09-42-01.png
Screenshot_from_2024-04-19_09-42-01.png
Attachment 32: Screenshot_from_2024-04-19_09-41-31.png
Screenshot_from_2024-04-19_09-41-31.png
Attachment 33: Screenshot_from_2024-04-19_09-40-02.png
Screenshot_from_2024-04-19_09-40-02.png
Attachment 34: Screenshot_from_2024-04-19_09-37-03.png
Screenshot_from_2024-04-19_09-37-03.png
Attachment 35: Screenshot_from_2024-04-19_09-46-35.png
Screenshot_from_2024-04-19_09-46-35.png
Attachment 36: Screenshot_from_2024-04-19_09-47-08.png
Screenshot_from_2024-04-19_09-47-08.png
Attachment 37: Screenshot_from_2024-04-19_10-44-19.png
Screenshot_from_2024-04-19_10-44-19.png
Attachment 38: Screenshot_from_2024-04-19_10-43-58.png
Screenshot_from_2024-04-19_10-43-58.png
  482   Sat Jun 18 09:46:57 2022 OH, NHFriday 17th June
Summary of the day:
Small improvements were made throughout the day but by far the biggest change to the rate in the FEEs came from disabling the waveforms
This was done by setting ADC con

Initial jumper configuration:
FEE LK
1   2 4
2   1 2 4
3   2 3 4
4   2 4
5   2 4
6   1 2 4
7   2 3 4
8   1 2 4

13:22 LK2, 3 and 4 on all p+n - attachment 1

14:20 AC mains relay connected to AC voltage stabilisers + on separate mains socket from DESPEC - attachment 2
      Same configuration but waveforms disabled (p+n improvment) - attachment 3
      Layout 1 - attachment 4

14:32 all FEE power cables moved to PSU 2 (Lower one in rack, upper one was slow to start for 1 FEE pair, likely the 5V slow to start)
      Previously all FEEs on their own pair within  a psu
      New order 1,3 2,4 5,7 6,8
      Waveforms on - all bad - attachment 5
      Waveforms off - all p+n better - attachment 6

14:54 interlock removed from AC mains relay
      Also noticed adaptor board on 2 not in fully -> Now in
      p+n generally better - attachment 7
      Without waveforms - p+n better again - attachment 8

16:04 Platform moved into position
      Interlock power cable removed from extension lead
      w/ waveform - p+n slightly worse
      w/o waveform - p+n same as beffore

16:14 LK3 removed from 1 and 5
      w/ waveform -> p+n slightly worse - attachment 9
      w/o waveform -> p+n best yet - attachment 10

16:33 LK1 wasn't on all n+n was only on 2, 6 and 8
      Removed from 6 -> only one LK1 per DSSD
      w/ waveforms - all bad - attachment 11
      w/o waveforms - no change - attachment 12

16:59 LK1 removed from 2->4 This is the FEE with the BIAS braid
      w/ waveform p+n better - attachment 13
      w/0 waveform p+n better, n+n worse? - attachment 14

17:21 Bias filters added to n+n
      w/ waveform - no change - attachment 15
      w/o wwaveofrm - P=n worse - attachment 16
      Stats 0x21 - attachment 17
      Stats 0x14 - attachment 18
      Stats 0x12 - attachment 19

17:42 Bias filter removed and bias floated (LK1 removed)
      w/ waveform all worse - attachment 20
      w/o waveform -> p+n worse, n+n similar

18:29 LK1 back on 4+8
      w/ waveform -> poor
      w/0 waveform - recovered previous - attachment 20


18:32 LK 2 and 4 removed from 2 +6
      Also noticed ground lemo on aida02 not fully inserted which has been corrected
      w/ waveform poor - attachment 21
      Threshold 0xF - attachment 22
      w/o waveform - attachment 23
Attachment 1: 220617_1322_stats.png
220617_1322_stats.png
Attachment 2: 220617_1420_stats.png
220617_1420_stats.png
Attachment 3: 220617_11428_stats.png
220617_11428_stats.png
Attachment 4: 220617_1428_layout1.png
220617_1428_layout1.png
Attachment 5: 220617_1440_stats_waveform_on.png
220617_1440_stats_waveform_on.png
Attachment 6: 220617_1444_stats_no_waveform.png
220617_1444_stats_no_waveform.png
Attachment 7: 220617_1459_stats_waveform_on.png
220617_1459_stats_waveform_on.png
Attachment 8: 220617_1517_stats_no_waveform.png
220617_1517_stats_no_waveform.png
Attachment 9: 220617_1624_stats_waveform_on.png
220617_1624_stats_waveform_on.png
Attachment 10: 220617_1626_stats_waveform_off.png
220617_1626_stats_waveform_off.png
Attachment 11: 220617_1649_stats_waveform_on.png
220617_1649_stats_waveform_on.png
Attachment 12: 220617_1652_no_waveform.png
220617_1652_no_waveform.png
Attachment 13: 220617_1709_waveform_on.png
220617_1709_waveform_on.png
Attachment 14: 220617_1724_stats_no_waveform.png
220617_1724_stats_no_waveform.png
Attachment 15: 220617_1729_stats_waveform_on.png
220617_1729_stats_waveform_on.png
Attachment 16: 220617_1733_stats_no_waveform.png
220617_1733_stats_no_waveform.png
Attachment 17: 220617_1736_0x21.png
220617_1736_0x21.png
Attachment 18: 220617_1738_0x14.png
220617_1738_0x14.png
Attachment 19: 220617_1738_0x12.png
220617_1738_0x12.png
Attachment 20: 220617_1807_stats_waveform_on.png
220617_1807_stats_waveform_on.png
Attachment 21: 220617_1832_stats_no_waveform.png
220617_1832_stats_no_waveform.png
Attachment 22: 220617_1844_0xf.png
220617_1844_0xf.png
Attachment 23: 220617_1845_stats_waveform_on.png
220617_1845_stats_waveform_on.png
  229   Fri Apr 16 07:19:00 2021 OH, MAFriday 16th April 08:00-16:00
08:19 System wide checks all ok
      Baselines reset on WR and FPGA checks for the start of the day

      Statistics ok - attachment 1
      Temperatures ok - attachment 2
      Bias and leakage currents ok - attachment 3

09:03 During the reset last night I repaired the OptionsDB for aida02 from the Options.Pristine directory.
      No corruptions have been noted since

10:10 System check, clcock and ADC ok

	 Base 		Current 	Difference
aida07 fault 	 0xfb3a : 	 0xfb3d : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA and memory check ok

      Statistics ok - attachment 4
      Temperatures ok - attachment 5
      Bias and leakage currents ok - attachment 6

10:39 Fission fragments were punching through AIDA
      This was even with the maxmimum degraders in place
      They are now placing a thick plate in front of AIDA to block the fragments while they beam tune.

11:38 They have found the issue with the degrader and have corrected it.
      We now see next to no implants in AIDA as expect.
      Histograms reset at around 11:35
      Layout 2 shows very few implants in 5 minutes - attachment 7

12:08 The high implantation rates of ions into AIDA is clearly seen in the large transients on the leakage currents - attachment 8
      This was a large amount of dose going into the detectors which we should try to avoid

13:25 system check, clock and ADC ok

	 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      Statistics ok - attachment 9
      Temperatures ok - attachment 10
      Bias and leakage currents ok - attachment 11
FPGA and memory check ok


16:10 system check, clock and ADC ok

	
		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb3f : 	 5  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


      Statistics ok - attachment 12
      Temperatures ok - attachment 13
      Bias and leakage currents ok - attachment 14
FPGA and memory check ok
Attachment 1: 210416_0820_Stats.png
210416_0820_Stats.png
Attachment 2: 210416_0821_Temp.png
210416_0821_Temp.png
Attachment 3: 210416_0824_Bias.png
210416_0824_Bias.png
Attachment 4: 210416_1003Stats.png
210416_1003Stats.png
Attachment 5: 210416_1007Temp.png
210416_1007Temp.png
Attachment 6: 210416_1019_voltages.png
210416_1019_voltages.png
Attachment 7: 210416_1139_layout2.png
210416_1139_layout2.png
Attachment 8: 210416_1207_Leakage.png
210416_1207_Leakage.png
Attachment 9: 210416_1311_Stats.png
210416_1311_Stats.png
Attachment 10: 210416_1312_Temp.png
210416_1312_Temp.png
Attachment 11: 210416_1315_Voltages.png
210416_1315_Voltages.png
Attachment 12: 210416_1412_Voltages.png
210416_1412_Voltages.png
Attachment 13: 210416_1548_Stats.png
210416_1548_Stats.png
Attachment 14: 210416_1553_Temp.png
210416_1553_Temp.png
  319   Fri May 14 19:01:09 2021 JSFriday 14th May 20:00-00:00
20:00 Taking over from CB

20:33 Doing full checks.
Temps ok elog:319/8  
Bias ok elog:319/1
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

FPGA Timestamp errors -
		 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Memory Info-
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     35     34     30      8      1      4      1      2      2      3      6   : 35772
aida02 :     37     30     28      5      3      1      2      4      2      3      6   : 36644
aida03 :     41     29     30     16      4      3      1      5      1      3      6   : 36588
aida04 :     13      6     23     12      4      4      4      4      2      2      6   : 35412
aida05 :     20     32     23     10      2      2      2      2      1      4      6   : 36736
aida06 :     14      5     10      5      1      3      2      3      2      3      6   : 35680
aida07 :     44     23     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     39     30     28      9      0      3      2      3      3      2      6   : 35308
aida09 :     41     31     28     10      2      5      1      3      1      3      6   : 35484
aida10 :     43     23     27      7      3      1      1      3      3      3      6   : 36916
aida11 :     39     35     17     10      3      3      2      2      2      3      6   : 35908
aida12 :     37     36     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     47     32     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     23     33     14     10      3      4      3      4      3      2      6   : 36164
aida15 :     44     28     24     12      3      2      4      3      3      2      6   : 35920
aida16 :     22     17     24     17      3      4      2      3      1      3      6   : 35648

Stats
Aida Correlation Info #8 - elog:319/2 same as elog:317/27 
Resume info #3 - elog:319/3 same as elog:317/26 
Pause info #2 - elog:319/4 same as elog:317/32 
AIDA disk info #6 - elog:319/5 same as elog:317/31 
AIDA ADC data items - elog:319/6 same as elog:317/30 
Good Events - elog:319/7 same as elog:317/29
[I had some artifacts in AnyDesk when looking at the stats pages with many zeros, but could see 
clearly once uploaded the screenshots] 

Analysis R5_108 elog:319/9
Pause 149 Resume 150
Highest deadtime FEE10 3%

21:01
ucesb ok - Max 1700 Hz Implant - 1MHz Decay

21:35
Temps ok
Bias ok
Clock check - ok
ADC Calc check - ok
White Rabbit - 
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc5 : 	 13  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     39     34     29      8      1      4      1      2      2      3      6   : 35772
aida02 :     46     29     28      5      3      2      2      3      2      3      6   : 36288
aida03 :     39     25     29     16      4      2      2      5      1      3      6   : 36660
aida04 :     14      6     19     12      4      4      3      4      2      2      6   : 35096
aida05 :     20     30     24     10      2      2      2      2      1      4      6   : 36736
aida06 :     26      4     11      5      2      4      3      2      2      3      6   : 35672
aida07 :     46     22     22      9      4      3      1      3      2      3      6   : 36200
aida08 :     28     32     28      9      0      2      2      4      3      2      6   : 35664
aida09 :     32     30     29     10      3      5      2      3      1      3      6   : 35776
aida10 :     39     23     25      7      3      1      1      3      3      3      6   : 36868
aida11 :     40     31     21     10      3      3      1      4      1      3      6   : 35688
aida12 :     17     31     12     10      2      2      3      4      3      2      6   : 35772
aida13 :     35     28     22      7      4      2      3      4      3      2      6   : 36012
aida14 :     30     33     14     10      3      4      3      4      3      2      6   : 36192
aida15 :     34     26     24     12      3      2      4      3      3      2      6   : 35864
aida16 :     38     24     27     17      3      4      2      3      1      3      6   : 35816

Stats
Good Events - elog:319/10
AIDA ADC data items - elog:319/11
AIDA disk info #6 - elog:319/12 
Pause info #2 - elog:319/13
Resume info #3 - elog:319/14 
Aida Correlation Info #8 - elog:319/15  - rates showed zero, checked again and normal elog:319/17

Analysis R5_128 elog:319/16
Pause 167 Resume 166
Highest deadtime FEE10 5%

22:00
Temps ok
Stats ok 
ucesb ok

22:35 
Temps ok
Stats ok 
ucesb- DSSD 2 is reading consistently 30% lower, not sure if this is dead time issue cropping up 
again elog:319/17

22:54 They are closing the file, we are on R5_155

23:00 
Attachment 1: bias1.png
bias1.png
Attachment 2: Screenshot_2021-05-14_Statistics_aidas-gsi(10).png
Screenshot_2021-05-14_Statistics_aidas-gsi(10).png
Attachment 3: Screenshot_2021-05-14_Statistics_aidas-gsi(9).png
Screenshot_2021-05-14_Statistics_aidas-gsi(9).png
Attachment 4: Screenshot_2021-05-14_Statistics_aidas-gsi(8).png
Screenshot_2021-05-14_Statistics_aidas-gsi(8).png
Attachment 5: Screenshot_2021-05-14_Statistics_aidas-gsi(7).png
Screenshot_2021-05-14_Statistics_aidas-gsi(7).png
Attachment 6: Screenshot_2021-05-14_Statistics_aidas-gsi(6).png
Screenshot_2021-05-14_Statistics_aidas-gsi(6).png
Attachment 7: Screenshot_2021-05-14_Statistics_aidas-gsi(5).png
Screenshot_2021-05-14_Statistics_aidas-gsi(5).png
Attachment 8: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: R5_108_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256861958 ( 1503007.9 Hz)
        Other data format:    5058042 (   29596.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        150 (       0.9 Hz)
                              SYNC100:      32636 (     191.0 Hz)
                              WR48-63:      32636 (     191.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4992471 (   29213.1 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     207149 (    1212.1 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      170.899 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                4.459               72.353
  2                1.217               86.117
  3                3.357               28.177
  4                0.000              160.970
  5                0.495               76.386
  6                0.216               93.617
  7                1.418               40.259
  8                3.253               44.571
  9                0.000              168.837
 10                5.250                9.058
 11                1.956               49.014
 12                0.182               82.483
 13                0.000              160.585
 14                0.143               88.740
 15                3.053               35.456
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8464368       3303          0          0          2          2       1138       1138          0       1023          0      11424
  1   16399642       4233          0          0         17         18       2099       2099          0          0          0      22505
  2   13045073    1576129          0          0          6          6       1793       1793          0    1572531          0      18179
  3   26563169    1558422          0          0         14         14       3487       3487          0    1551420          0      26353
  4    2213090     510296          0          0          0          0        328        328          0     509640          0       5100
  5   15498485     314714          0          0          4          4       1986       1986          0     310734          0      15610
  6   12553775     609520          0          0          7          7       1573       1573          0     606360          0      10360
  7   23384876     446841          0          0         10         10       3029       3029          0     440763          0      18175
  8   21832531       5542          0          0         15         15       2756       2756          0          0          0      22662
  9    1045474        278          0          0          0          0        139        139          0          0          0       1866
 10   42357744      10552          0          0         43         43       5233       5233          0          0          0      12639
 11   20606319       4998          0          0         11         11       2488       2488          0          0          0       4351
 12   14311836       3576          0          0          6          6       1782       1782          0          0          0      20348
 13    3016503        780          0          0          0          0        390        390          0          0          0       1508
 14   13047482       3248          0          0          4          4       1620       1620          0          0          0      12826
 15   22521591       5610          0          0         10         10       2795       2795          0          0          0       3243
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:74697.445s (    0.428 blocks/s,   0.027 Mb/s)
Attachment 10: Screenshot_2021-05-14_Statistics_aidas-gsi(16).png
Screenshot_2021-05-14_Statistics_aidas-gsi(16).png
Attachment 11: Screenshot_2021-05-14_Statistics_aidas-gsi(15).png
Screenshot_2021-05-14_Statistics_aidas-gsi(15).png
Attachment 12: Screenshot_2021-05-14_Statistics_aidas-gsi(14).png
Screenshot_2021-05-14_Statistics_aidas-gsi(14).png
Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi(13).png
Screenshot_2021-05-14_Statistics_aidas-gsi(13).png
Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(12).png
Screenshot_2021-05-14_Statistics_aidas-gsi(12).png
Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(11).png
Screenshot_2021-05-14_Statistics_aidas-gsi(11).png
Attachment 16: R5_128_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256841478 ( 1482693.9 Hz)
        Other data format:    5078522 (   29317.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        167 (       1.0 Hz)
                               RESUME:        166 (       1.0 Hz)
                              SYNC100:      32644 (     188.4 Hz)
                              WR48-63:      32644 (     188.4 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5012901 (   28938.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     205165 (    1184.4 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      173.226 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.621               76.080
  2                1.556               81.876
  3                3.719               27.628
  4                0.000              162.954
  5                0.710               80.266
  6                0.170               93.769
  7                2.714               44.740
  8                3.483               45.565
  9                0.000              167.005
 10                8.522               10.948
 11                2.181               50.562
 12                0.202               85.035
 13                0.000              161.510
 14                0.146               88.959
 15                1.812               38.597
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8468806       3273          0          0          7          7       1112       1112          0       1035          0      11260
  1   16364560       4360          0          0         13         13       2167       2167          0          0          0      23505
  2   13265609    1594590          0          0          6          6       1881       1881          0    1590816          0      18254
  3   26054686    1576337          0          0         14         14       3422       3422          0    1569465          0      26152
  4    2223261     507850          0          0          0          0        362        362          0     507126          0       4952
  5   15152054     312213          0          0          8          7       1938       1938          0     308322          0      15336
  6   12659379     606961          0          0          4          4       1612       1612          0     603729          0      10099
  7   22669666     438118          0          0         14         14       2841       2841          0     432408          0      17594
  8   22337729       5564          0          0         15         15       2767       2767          0          0          0      22524
  9    1020352        240          0          0          0          0        120        120          0          0          0       1807
 10   42596491      10676          0          0         59         59       5279       5279          0          0          0      12076
 11   20672887       5060          0          0         13         13       2517       2517          0          0          0       4379
 12   14335521       3608          0          0          5          5       1799       1799          0          0          0      20105
 13    2812573        690          0          0          0          0        345        345          0          0          0       1405
 14   13325633       3260          0          0          4          4       1626       1626          0          0          0      12554
 15   22882271       5722          0          0          5          5       2856       2856          0          0          0       3163
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:78055.344s (    0.410 blocks/s,   0.026 Mb/s)
Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(17).png
Screenshot_2021-05-14_Statistics_aidas-gsi(17).png
Attachment 18: ucesb.png
ucesb.png
Attachment 19: _~.png
_~.png
Attachment 20: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 21: Screenshot_2021-05-14_Statistics_aidas-gsi(23).png
Screenshot_2021-05-14_Statistics_aidas-gsi(23).png
Attachment 22: Screenshot_2021-05-14_Statistics_aidas-gsi(22).png
Screenshot_2021-05-14_Statistics_aidas-gsi(22).png
Attachment 23: Screenshot_2021-05-14_Statistics_aidas-gsi(21).png
Screenshot_2021-05-14_Statistics_aidas-gsi(21).png
Attachment 24: Screenshot_2021-05-14_Statistics_aidas-gsi(20).png
Screenshot_2021-05-14_Statistics_aidas-gsi(20).png
Attachment 25: Screenshot_2021-05-14_Statistics_aidas-gsi(19).png
Screenshot_2021-05-14_Statistics_aidas-gsi(19).png
Attachment 26: Screenshot_2021-05-14_Statistics_aidas-gsi(18).png
Screenshot_2021-05-14_Statistics_aidas-gsi(18).png
Attachment 27: R5_157_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256650793 ( 1485751.4 Hz)
        Other data format:    5269207 (   30503.4 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        130 (       0.8 Hz)
                               RESUME:        129 (       0.7 Hz)
                              SYNC100:      32643 (     189.0 Hz)
                              WR48-63:      32643 (     189.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5203662 (   30124.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     223116 (    1291.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      172.741 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.094               74.418
  2                0.927               76.576
  3                3.655               26.911
  4                0.000              161.752
  5                0.023               80.653
  6                0.215               91.583
  7                1.871               44.002
  8                1.038               39.676
  9                0.000              166.718
 10                8.591               10.747
 11                3.917               45.280
 12                0.214               88.240
 13                0.000              163.408
 14                0.280               88.718
 15                1.416               44.122
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8604841       3308          0          0          0          0       1138       1138          0       1032          0      12424
  1   16359820       4228          0          0         13         13       2101       2101          0          0          0      24974
  2   13547296    1610585          0          0          6          6       1930       1930          0    1606713          0      19730
  3   25981004    1589167          0          0         14         13       3491       3491          0    1582158          0      28914
  4    2316430     549243          0          0          0          0        351        351          0     548541          0       5271
  5   15208801     341870          0          0          1          1       1974       1974          0     337920          0      16457
  6   12807457     657341          0          0          5          5       1667       1667          0     653997          0      10925
  7   22697274     479019          0          0         10         10       2849       2849          0     473301          0      19398
  8   22587880       5594          0          0          5          5       2792       2792          0          0          0      24704
  9    1078064        266          0          0          0          0        133        133          0          0          0       1963
 10   42543713      10740          0          0         43         43       5327       5327          0          0          0      13313
 11   20362862       5182          0          0         14         14       2577       2577          0          0          0       4606
 12   13994455       3436          0          0          7          7       1711       1711          0          0          0      21818
 13    2731986        666          0          0          0          0        333        333          0          0          0       1621
 14   13145005       3222          0          0          8          8       1603       1603          0          0          0      13539
 15   22683905       5340          0          0          4          4       2666       2666          0          0          0       3459
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:83137.008s (    0.385 blocks/s,   0.024 Mb/s)
  317   Fri May 14 14:57:32 2021 CB + OH, TD, NHFriday 14th May 16:00-20:00
16:00 Shift taken over from CA

16:00 
Temperatures OK. Attach 1. 
Upon reloading aida11 first returns "No respose" for all values. 
Upon refreshing, aida11 responds but aida 9 returns "No response". 
Upon refreshing again, attach 1.

Bias OK. Attach 2.

Statistics OK. Attach 3-7

System-wide checks
Clock OK
ADC - all fail
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     32     28     25      7      1      4      1      3      2      3      6   : 36112
aida02 :     43     28     29      5      4      2      3      4      1      3      6   : 36092
aida03 :     39     26     28      5      4      3      2      5      1      3      6   : 36428
aida04 :      9      5     21      8      5      4      4      4      2      2      6   : 35292
aida05 :     46     32     26      7      2      2      2      2      2      3      6   : 35768
aida06 :      4      5     16      1      3      4      2      4      1      3      6   : 35352
aida07 :     47     21     25      5      4      3      1      4      1      3      6   : 35604
aida08 :     62     33     32      6      0      2      3      3      3      2      6   : 35520
aida09 :     33     25     28      7      3      5      2      4      1      3      6   : 36140
aida10 :     47     29     22      6      3      2      2      2      2      3      6   : 35716
aida11 :     39     30     29      6      2      3      1      3      2      3      6   : 36124
aida12 :     38     36     20      6      3      2      3      3      3      2      6   : 35448
aida13 :     42     29     24      3      4      2      3      3      3      2      6   : 35440
aida14 :     31     28     18      7      3      3      4      4      2      2      6   : 35228
aida15 :     48     25     30      5      3      2      4      4      2      2      6   : 35272
aida16 :     37     16     28      6      3      3      2      2      2      3      6   : 35796

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021



16:15 Control handed over to OH temporarily.
16:25 AIDA stopped. OH attempts to update Merger. Discriminators off.
16:29 AIDA restart begins.
16:33 AIDA restarted.
16:40 Dead time still very high ~50% in most FEEs.
16:50 Checking effect of threshold on dead time. 
16:53 Checking waveform effect on deadtime. DAQ stopped. Turning waveforms on, threshold increased to max. This
*reduces* deadtime to <10%.
      Sampling ADCs were disabled (switched off) late Wednesday evening.
17:01 DAQ can be restarted.
17:27 Finished checking effect of threshold on dead time. 120 keV front. 320 keV back.
17:33 Analysis of R5_45 - x strips 0xc - y strips 0x20



17:40 OH hands back control.
Temperature OK - attach 10
Bias OK - attach 11
Stats - attach 12-17

System-wide checks
Clock - all pass
ADC - all pass
WR
	 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     56     35     27      8      1      4      1      2      2      3      6   : 35816
aida02 :     43     31     27      5      3      1      3      3      2      3      6   : 36404
aida03 :     45     28     28     16      4      2      2      4      1      3      6   : 36180
aida04 :      2      2     18     12      4      3      3      4      2      2      6   : 34872
aida05 :     28     28     24     10      2      2      2      2      1      4      6   : 36752
aida06 :     28     11      9      5      2      3      3      3      1      3      6   : 35064
aida07 :     52     24     23      9      4      3      1      2      2      3      6   : 35744
aida08 :     21     26     27      8      0      2      3      3      3      2      6   : 35284
aida09 :     40     32     26     10      3      5      1      3      1      3      6   : 35520
aida10 :     46     23     27      7      3      1      1      3      3      3      6   : 36928
aida11 :     49     31     20     10      2      2      1      2      2      3      6   : 35516
aida12 :     39     35     17      9      2      2      3      4      3      2      6   : 35940
aida13 :     40     32     20      7      4      2      3      4      3      2      6   : 36032
aida14 :     33     33     16     10      3      4      3      4      3      2      6   : 36236
aida15 :     53     27     25     12      3      2      4      4      2      2      6   : 35452
aida16 :     35     22     32     15      3      4      3      3      1      3      6   : 36060

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

18:30
Temperatures - OK (attach 18)
Bias OK (attach 19)
Stats - attach 20-24

System-wide checks
Clock - all pass
ADC - all pass
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     31     31      8      1      4      1      2      2      3      6   : 35788
aida02 :     42     31     29      5      3      2      4      4      1      3      6   : 36304
aida03 :     29     30     29     16      4      3      2      4      1      3      6   : 36276
aida04 :     28      4     20     12      4      3      4      4      2      2      6   : 35280
aida05 :     25     28     23     10      2      2      2      2      1      4      6   : 36724
aida06 :      5      3     10      5      2      4      2      2      2      3      6   : 35308
aida07 :     38     26     22      9      4      3      1      2      2      3      6   : 35688
aida08 :     52     30     28      8      0      2      2      3      3      2      6   : 35200
aida09 :     22     26     21     10      3      4      1      4      1      3      6   : 35704
aida10 :     37     28     25      7      3      1      1      3      3      3      6   : 36900
aida11 :     53     30     20     10      2      3      2      2      2      3      6   : 35908
aida12 :     37     34     17     10      2      2      3      4      3      2      6   : 35956
aida13 :     47     34     16      7      4      2      3      4      3      2      6   : 36012
aida14 :     37     29     17     10      3      4      3      4      3      2      6   : 36236
aida15 :     44     28     25     12      3      2      4      4      2      2      6   : 35424
aida16 :     40     21     33     15      3      4      1      3      1      3      6   : 35576

18:40 Analysed R5_67. Attached. Dead time improved. Max ~1.5%


19:40 Temperatures OK
Bias OK - attach 26
Stats - attach 27-32

System-wide checks

Clock, ADC - all pass
WR

	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf934 : 	 2  
aida02 fault 	 0x62ec : 	 0x62ee : 	 2  
aida03 fault 	 0x8679 : 	 0x867b : 	 2  
aida04 fault 	 0xf0e4 : 	 0xf0e6 : 	 2  
aida05 fault 	 0x9db8 : 	 0x9dc4 : 	 12  
aida06 fault 	 0x7f18 : 	 0x7f1a : 	 2  
aida07 fault 	 0xdd2c : 	 0xdd2e : 	 2  
aida08 fault 	 0x1557 : 	 0x1559 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     41     33     30      8      1      4      1      2      2      3      6   : 35788
aida02 :     51     30     30      5      3      2      3      4      1      3      6   : 36092
aida03 :     52     31     28     16      4      2      2      5      1      3      6   : 36744
aida04 :      6      8     20     12      4      4      3      4      2      2      6   : 35096
aida05 :     30     31     22     10      2      2      2      2      1      4      6   : 36752
aida06 :     12      6     10      5      2      3      3      2      2      3      6   : 35488
aida07 :     53     24     21      9      4      3      1      3      2      3      6   : 36228
aida08 :     36     28     28      9      0      2      3      3      3      2      6   : 35408
aida09 :     27     28     25     10      3      4      2      3      1      3      6   : 35548
aida10 :     50     23     26      7      3      1      1      3      3      3      6   : 36928
aida11 :     42     28     22     10      3      3      2      2      2      3      6   : 35944
aida12 :     39     35     15     10      2      2      2      4      3      2      6   : 35684
aida13 :     43     34     17      7      4      2      3      4      3      2      6   : 36012
aida14 :     42     32     16     10      3      4      3      4      3      2      6   : 36264
aida15 :     36     30     25     12      3      2      4      3      3      2      6   : 35920
aida16 :     40     21     25     15      3      4      2      3      1      3      6   : 35704

	
 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

19:40 Analysed R5_86. Attached. Dead time as before.

19:57 Shift handed over to James S
Attachment 1: Temp1.png
Temp1.png
Attachment 2: Bias1.png
Bias1.png
Attachment 3: Stats1-1.png
Stats1-1.png
Attachment 4: Stats1-2.png
Stats1-2.png
Attachment 5: Stats1-3.png
Stats1-3.png
Attachment 6: Stats1-4.png
Stats1-4.png
Attachment 7: Stats1-5.png
Stats1-5.png
Attachment 8: Stats1-6.png
Stats1-6.png
Attachment 9: R5_45_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  257173968 ( 1605341.2 Hz)
        Other data format:    4746032 (   29625.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        149 (       0.9 Hz)
                               RESUME:        149 (       0.9 Hz)
                              SYNC100:      32597 (     203.5 Hz)
                              WR48-63:      32597 (     203.5 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4680540 (   29217.0 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     198582 (    1239.6 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      160.199 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.826               60.560
  2                1.857               72.569
  3                3.101               18.747
  4                0.000              149.092
  5                0.725               60.979
  6                0.276               85.413
  7                1.668               27.289
  8                2.719               31.235
  9                0.000              157.899
 10                6.867               10.073
 11                2.634               42.733
 12                0.375               69.093
 13                0.000              148.656
 14                0.255               79.878
 15                3.499               32.266
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8318308       3120          0          0          3          3       1077       1077          0        960          0      11152
  1   17482671       4536          0          0         10         10       2258       2258          0          0          0      22118
  2   12591028    1472109          0          0          7          7       1790       1790          0    1468515          0      17237
  3   26467090    1465014          0          0         17         17       3574       3574          0    1457832          0      25917
  4    2005748     477188          0          0          0          0        331        331          0     476526          0       4838
  5   16244903     298514          0          0          5          5       2012       2012          0     294480          0      14514
  6   11816116     567974          0          0          7          7       1554       1554          0     564852          0       9683
  7   24437155     423701          0          0          8          8       3155       3155          0     417375          0      17498
  8   22741388       5724          0          0         10         10       2852       2852          0          0          0      21729
  9    1027419        274          0          0          0          0        137        137          0          0          0       1781
 10   39497864       9636          0          0         47         47       4771       4771          0          0          0      12114
 11   19835466       4892          0          0          8          8       2438       2438          0          0          0       4107
 12   15299334       3708          0          0          8          8       1846       1846          0          0          0      19448
 13    3576125        840          0          0          0          0        420        420          0          0          0       1437
 14   13120071       3182          0          0          7          7       1584       1584          0          0          0      12115
 15   22713282       5620          0          0         12         12       2798       2798          0          0          0       2894
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:63108.355s (    0.507 blocks/s,   0.032 Mb/s)
Attachment 10: Temps2.png
Temps2.png
Attachment 11: Bias2.png
Bias2.png
Attachment 12: Stats2-1.png
Stats2-1.png
Attachment 13: Stats2-2.png
Stats2-2.png
Attachment 14: Stats2-3.png
Stats2-3.png
Attachment 15: Stats2-4.png
Stats2-4.png
Attachment 16: Stats2-5.png
Stats2-5.png
Attachment 17: Stats2-6.png
Stats2-6.png
Attachment 18: Temp3.png
Temp3.png
Attachment 19: Bias3.png
Bias3.png
Attachment 20: Stats3-1.png
Stats3-1.png
Attachment 21: Stats3-2.png
Stats3-2.png
Attachment 22: Stats3-3.png
Stats3-3.png
Attachment 23: Stats3-4.png
Stats3-4.png
Attachment 24: Stats3-5.png
Stats3-5.png
Attachment 25: R5_67_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  255938817 ( 1305497.2 Hz)
        Other data format:    5981183 (   30508.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        114 (       0.6 Hz)
                               RESUME:        113 (       0.6 Hz)
                              SYNC100:      32730 (     166.9 Hz)
                              WR48-63:      32730 (     166.9 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    5915496 (   30173.9 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     257806 (    1315.0 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      196.047 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                2.111              102.399
  2                0.513              105.076
  3                3.587               39.451
  4                0.000              182.105
  5                0.792              103.780
  6                0.167              115.991
  7                1.244               72.680
  8                0.805               56.828
  9                0.000              190.127
 10                2.629                6.967
 11                0.295               76.751
 12                0.222              114.996
 13                0.000              188.225
 14                0.031              124.690
 15                2.491               63.947
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8223770       3364          0          0          2          2       1092       1092          0       1176          0      13825
  1   16628095       4338          0          0         12         12       2157       2157          0          0          0      28636
  2   13173045    1831289          0          0          3          3       1864       1864          0    1827555          0      22667
  3   27251797    1807380          0          0         13         13       3668       3668          0    1800018          0      33190
  4    2564061     622927          0          0          0          0        404        404          0     622119          0       6108
  5   14810679     385418          0          0          6          6       1903       1903          0     381600          0      19053
  6   12067625     742516          0          0          6          6       1616       1616          0     739272          0      12600
  7   21266971     549224          0          0         11         11       2723       2723          0     543756          0      22659
  8   23290358       5786          0          0          5          5       2888       2888          0          0          0      28199
  9    1157906        324          0          0          0          0        162        162          0          0          0       2443
 10   45608185      11489          0          0         27         26       5718       5718          0          0          0      16139
 11   20146427       4876          0          0          6          6       2432       2432          0          0          0       5579
 12   13411417       3300          0          0          7          7       1643       1643          0          0          0      24922
 13    2681820        654          0          0          0          0        327        327          0          0          0       1925
 14   11884997       2904          0          0          1          1       1451       1451          0          0          0      15892
 15   21771664       5394          0          0         15         15       2682       2682          0          0          0       3969
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:67311.180s (    0.475 blocks/s,   0.030 Mb/s)
Attachment 26: Stats_RESUME.png
Stats_RESUME.png
Attachment 27: Stats-CORRELATION.png
Stats-CORRELATION.png
Attachment 28: Bias4.png
Bias4.png
Attachment 29: Stats-good.png
Stats-good.png
Attachment 30: Stats-AIDAADC.png
Stats-AIDAADC.png
Attachment 31: Stats_DISC.png
Stats_DISC.png
Attachment 32: Stats_PAUSE.png
Stats_PAUSE.png
Attachment 33: R5_86_analysis.txt
 *** TDR format 3.3.0 analyser - TD - January 2019
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  256871525 ( 1549964.4 Hz)
        Other data format:    5048475 (   30462.5 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        133 (       0.8 Hz)
                               RESUME:        132 (       0.8 Hz)
                              SYNC100:      32617 (     196.8 Hz)
                              WR48-63:      32617 (     196.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    4982976 (   30067.3 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     217412 (    1311.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      165.727 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  1                1.819               66.425
  2                0.365               78.785
  3                3.792               29.131
  4                0.000              149.995
  5                0.868               74.751
  6                0.080               85.583
  7                2.138               33.988
  8                2.899               39.410
  9                0.000              164.566
 10                6.499               11.006
 11                1.139               43.564
 12                0.944               80.601
 13                0.000              152.925
 14                0.102               87.597
 15                2.546               36.680
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    8622288       3316          0          0          4          3       1158       1158          0        993          0      11812
  1   17129008       4510          0          0         12         12       2243       2243          0          0          0      24429
  2   13474006    1553973          0          0          6          6       1869       1869          0    1550223          0      19028
  3   26129234    1524704          0          0         20         20       3374       3374          0    1517916          0      27835
  4    2168475     521045          0          0          0          0        352        352          0     520341          0       5247
  5   15823578     326080          0          0          4          4       1963       1963          0     322146          0      16074
  6   12515224     622754          0          0          2          2       1625       1625          0     619500          0      10516
  7   23990039     458079          0          0         12         12       3099       3099          0     451857          0      18924
  8   22108276       5660          0          0         13         13       2817       2817          0          0          0      24206
  9    1062355        236          0          0          0          0        118        118          0          0          0       1949
 10   41223894      10244          0          0         40         40       5082       5082          0          0          0      13136
 11   20572413       4974          0          0          7          7       2480       2480          0          0          0       4577
 12   14036482       3446          0          0          3          3       1720       1720          0          0          0      21370
 13    3191210        818          0          0          0          0        409        409          0          0          0       1573
 14   12949121       3198          0          0          3          3       1596       1596          0          0          0      13450
 15   21875922       5438          0          0          7          7       2712       2712          0          0          0       3286
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:70812.570s (    0.452 blocks/s,   0.028 Mb/s)
  316   Fri May 14 07:05:08 2021 CAFriday 14th May 08:00 - 16:00 shift
08:00 CA takes over

08:28 Problem at UNILAC - no beam

09:30 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveforms)

      WR decoder status:

		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA timestamp errors:

			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

09:36 FEE64 Temperatures ok - attachment 1

      Detector bias / leakage currents ok - attachment 2

      Statistics - attachments 3-8

09:50 beam is back - writing to file R4_265

09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA

10:10 FRS increase degrader thickness

10.28 all histograms, stats and merger stats zero'd

10.50 all system wide checks ok *except*
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc2 : 	 10  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      3      2      2      2      2      2      2      4      3      3      6   : 36860
aida02 :      7      3      5      1      2      5      2      4      3      3      6   : 37284
aida03 :      7      5      3      1      4      4      2      3      3      3      6   : 36756
aida04 :      3      1     20      4      2      3      3      5      1      3      6   : 36052
aida05 :     19     25     26      6      2      2      3      2      2      3      6   : 35828
aida06 :      1      3     16      1      3      4      1      5      1      3      6   : 35580
aida07 :      6      4      1      2      3      3      2      3      3      3      6   : 36552
aida08 :      8      5      2      3      2      3      2      2      2      4      6   : 37064
aida09 :      9      7      1      2      2      4      2      3      3      3      6   : 36652
aida10 :      2      5      3      1      1      4      2      3      3      3      6   : 36544
aida11 :     16      4      4      2      1      2      2      3      3      3      6   : 36384
aida12 :      2      1      1      3      1      2      2      3      3      3      6   : 36288
aida13 :      0      1      1      2      0      4      3      2      3      3      6   : 36184
aida14 :      6      3      2      0      2      3      2      3      3      3      6   : 36432
aida15 :     25      8      2      1      2      3      1      2      2      4      6   : 36836
aida16 :      3      5      1      2      2      2      3      2      3      3      6   : 36100


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Thu May 13 05:49:42 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

10.50 DAQ continues S496/R4_304

11:18 FEE64 Temperatures ok - attachment 9

      Detector bias & leakage currents ok - attachment 10

      statistics - attachments 11-16

15:00 FEE64           avg. CPU usage (%)

      1               52
      2               55
      3               55
      4               60
      5               93
      6               50
      7               53
      8               55
      9               55
      10              51
      11              50
      12              67
      13              56
      14              55
      15              55
      16              58

15:08 all system wide checks ok *except*

      all FEE64 fail ADC calibration (no waveform)

      WR decoder status:

      Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc3 : 	 11  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

    FPGA errors:

 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x3 : 	 3  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

15:10 Temperatures ok - attachment 17

      Detector bias / leakage current ok - attachment 18

      Statistics - attachments 18 -24
Attachment 1: Screenshot_from_2021-05-14_08-32-16.png
Screenshot_from_2021-05-14_08-32-16.png
Attachment 2: Screenshot_from_2021-05-14_08-32-45.png
Screenshot_from_2021-05-14_08-32-45.png
Attachment 3: Screenshot_from_2021-05-14_08-33-11.png
Screenshot_from_2021-05-14_08-33-11.png
Attachment 4: Screenshot_from_2021-05-14_08-33-33.png
Screenshot_from_2021-05-14_08-33-33.png
Attachment 5: Screenshot_from_2021-05-14_08-34-12.png
Screenshot_from_2021-05-14_08-34-12.png
Attachment 6: Screenshot_from_2021-05-14_08-34-44.png
Screenshot_from_2021-05-14_08-34-44.png
Attachment 7: Screenshot_from_2021-05-14_08-35-17.png
Screenshot_from_2021-05-14_08-35-17.png
Attachment 8: Screenshot_from_2021-05-14_08-35-55.png
Screenshot_from_2021-05-14_08-35-55.png
Attachment 9: Screenshot_from_2021-05-14_10-17-25.png
Screenshot_from_2021-05-14_10-17-25.png
Attachment 10: Screenshot_from_2021-05-14_10-17-43.png
Screenshot_from_2021-05-14_10-17-43.png
Attachment 11: Screenshot_from_2021-05-14_10-22-39.png
Screenshot_from_2021-05-14_10-22-39.png
Attachment 12: Screenshot_from_2021-05-14_10-23-13.png
Screenshot_from_2021-05-14_10-23-13.png
Attachment 13: Screenshot_from_2021-05-14_10-23-41.png
Screenshot_from_2021-05-14_10-23-41.png
Attachment 14: Screenshot_from_2021-05-14_10-24-12.png
Screenshot_from_2021-05-14_10-24-12.png
Attachment 15: Screenshot_from_2021-05-14_10-24-37.png
Screenshot_from_2021-05-14_10-24-37.png
Attachment 16: Screenshot_from_2021-05-14_10-25-20.png
Screenshot_from_2021-05-14_10-25-20.png
Attachment 17: Screenshot_from_2021-05-14_14-07-48.png
Screenshot_from_2021-05-14_14-07-48.png
Attachment 18: Screenshot_from_2021-05-14_14-11-23.png
Screenshot_from_2021-05-14_14-11-23.png
Attachment 19: Screenshot_from_2021-05-14_14-13-21.png
Screenshot_from_2021-05-14_14-13-21.png
Attachment 20: Screenshot_from_2021-05-14_14-13-38.png
Screenshot_from_2021-05-14_14-13-38.png
Attachment 21: Screenshot_from_2021-05-14_14-14-12.png
Screenshot_from_2021-05-14_14-14-12.png
Attachment 22: Screenshot_from_2021-05-14_14-14-39.png
Screenshot_from_2021-05-14_14-14-39.png
Attachment 23: Screenshot_from_2021-05-14_14-15-06.png
Screenshot_from_2021-05-14_14-15-06.png
Attachment 24: Screenshot_from_2021-05-14_14-15-26.png
Screenshot_from_2021-05-14_14-15-26.png
  315   Thu May 13 23:58:05 2021 OHFriday 14th May 00:00 - 08:00
00:00 OH Takes over.
 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

OK
0x210 

 p+n Thresholds 0xa (100keV)
 n+n Threshold 0x20 (320keV)

01:00 System wide checks
 Clock status ok
 ADC calibration fail as expected (Waveforms off)
 WR Errors
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dbf : 	 7  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 FPGA Errors
	
                Base Current Difference
aida05 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 Statistics ok - attachment 1
 Temp ok - attachment 2
 Bias and leakage current ok - attachment 3

01:42 Comparing the CS of AIDA3 and 4
      Stats have been running for a while (Believe since 5am)
      Looking at the counter ratio - attachment 4
      We still have a deadtime of around 14% in FEE4 (Assuming FEE3 has no dead time)

01:56 Noticed /media/ThirdDrive was at 97% full
      Let DESPEC locals know I would like to use the next run stop to change the directory.
      They said they could do it now so they stopped their run.
      Changed the symbolic link to /media/1e2....
      Now writing to R4_

03:26 MBS DAQ Stopped as beam stopped
03:33 Beam back now and DAQ running again

04:44 System wide checks as before
      Statistics ok - attachment 5
      Temperatures ok - attachment - 7
      Bias and leakage currents ok - attachment 6

05:53 MBS stops run as beam drops
06:00 MBS started a new run as beam is back

07:37 System wide checks clock ok and FPGA as before
      WR additional difference on FEE5
	
		 Base 		Current 	Difference
aida01 fault 	 0xf932 : 	 0xf933 : 	 1  
aida02 fault 	 0x62ec : 	 0x62ed : 	 1  
aida03 fault 	 0x8679 : 	 0x867a : 	 1  
aida04 fault 	 0xf0e4 : 	 0xf0e5 : 	 1  
aida05 fault 	 0x9db8 : 	 0x9dc0 : 	 8  
aida06 fault 	 0x7f18 : 	 0x7f19 : 	 1  
aida07 fault 	 0xdd2c : 	 0xdd2d : 	 1  
aida08 fault 	 0x1557 : 	 0x1558 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics ok - attachment 8
     Temp ok - attachment 9
     Bias and leakage ok - attachment 10
 
Attachment 1: 210514_0100_Stats.png
210514_0100_Stats.png
Attachment 2: 210514_0101_Temp.png
210514_0101_Temp.png
Attachment 3: 210514_0102_Bias.png
210514_0102_Bias.png
Attachment 4: 210514_0141_CS_Stats.png
210514_0141_CS_Stats.png
Attachment 5: 210514_0442_Stats.png
210514_0442_Stats.png
Attachment 6: 210514_0443_Bias.png
210514_0443_Bias.png
Attachment 7: 210514_0443_Temp.png
210514_0443_Temp.png
Attachment 8: 210514_0737_Stats.png
210514_0737_Stats.png
Attachment 9: 210514_0738_Temp.png
210514_0738_Temp.png
Attachment 10: 210514_0739_Bias.png
210514_0739_Bias.png
Attachment 11: Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi.png
Attachment 12: Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-14_Spectrum_Browser_aidas-gsi(1).png
Attachment 13: Screenshot_2021-05-14_Statistics_aidas-gsi.png
Screenshot_2021-05-14_Statistics_aidas-gsi.png
Attachment 14: Screenshot_2021-05-14_Statistics_aidas-gsi(1).png
Screenshot_2021-05-14_Statistics_aidas-gsi(1).png
Attachment 15: Screenshot_2021-05-14_Statistics_aidas-gsi(2).png
Screenshot_2021-05-14_Statistics_aidas-gsi(2).png
Attachment 16: Screenshot_2021-05-14_Statistics_aidas-gsi(3).png
Screenshot_2021-05-14_Statistics_aidas-gsi(3).png
Attachment 17: Screenshot_2021-05-14_Statistics_aidas-gsi(4).png
Screenshot_2021-05-14_Statistics_aidas-gsi(4).png
Attachment 18: Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-14_Temperature_and_status_scan_aidas-gsi.png
Attachment 19: Screenshot_2021-05-14_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-14_AIDA_Alerting_-_Grafana.png
Attachment 20: 600.png
600.png
Attachment 21: 601.png
601.png
  148   Fri Mar 13 08:01:53 2020 DKFriday 13th 8:00 to 16:00 shift
8:00 - See previous shift's elog entry.  All good.

9:00
    All system wide checks passed
    Merger okay, 3 Mega items / sec, all nnaida twinkle or stay lit
    Tape Server okay - 27 MB/s
    Histograms of implants look normal, about 5 counts per bin in SSD1 and SSD2 - see attachment #1
    Stats as usual - attachment #2
    Temps normal - attachment #3
    Biases and leaks fine -  attachment #4

Disk usage projection
    Space remaining: ~2 TB
    Fill rate: ~2 GB / 1 minute
    -> 1000 minutes =~ 17 hours

Still gzipping from R6_823

10:13 I noticed that the integer for the TapeServer UI feedbackk has been exceeded.
     	Kbytes Written:  -1986382400	Kbytes/sec:  26816
        Rate is fixed at that value on update.
        However, actual data files are being updated so data has been coming.
        DK and NH consulted and believe the situation is fine.
        We agreed next time the DESPEC team closes a run and starts a new one, they inform DK so he can restart the TapeServer for posterity and peace of mind.

10:25 Now we are on R10 after start and stop the TapeServer.  Problem is resolved.

11:30
      Implants: attachment #5
      Stats: attachment #6
      Biases / leaks: attachment #7
      Temps: attachment #8
      System wide checks performed, situation normal.
Attachment 1: daid-13-histosA.png
daid-13-histosA.png
Attachment 2: daid-13-statsA.png
daid-13-statsA.png
Attachment 3: daid-13-tempsA.png
daid-13-tempsA.png
Attachment 4: daid-13-biasA.png
daid-13-biasA.png
Attachment 5: daid-13-implantsB.png
daid-13-implantsB.png
Attachment 6: daid-13-statsB.png
daid-13-statsB.png
Attachment 7: daid-13-biasesB.png
daid-13-biasesB.png
Attachment 8: daid-13-tempsB.png
daid-13-tempsB.png
  146   Thu Mar 12 23:04:24 2020 LS, DKFriday 13th 00:00 to 08:00 shift
   Shift file starting R9_639

   system wide checks okay
   FEE temperatures okay (attachment 1)
   leakage currents normal and recorded to spreadsheet (attachment 2)
   good event stats are normal (attachment 3)
   merger running at 3.5 M items/sec
   tape running at 26 MB/sec
   
   rates histogram (attachment 4)
   low energy histograms (attachment 5 and 6)
   high energy histograms (attachment 7 and 8)

02.02 
   system wide checks okay
   FEE temperatures okay (attachment 9)
   leakage currents normal and recorded to spreadsheet (attachment 10)
   good event stats are normal (attachment 11)
   merger running at 3.5 M items/sec
   tape running at 27 MB/sec

04.09 
   system wide checks okay
   FEE temperatures okay (attachment 12)
   leakage currents normal and recorded to spreadsheet (attachment 13)
   good event stats are normal (attachment 14)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
   writing to file R9_840

06.07 
   system wide checks okay
   FEE temperatures okay (attachment 15)
   leakage currents normal and recorded to spreadsheet (attachment 16)
   good event stats are normal (attachment 17)
   merger running at 3 M items/sec
   tape running at 27 MB/sec
 
07.59 
   system wide checks okay
   FEE temperatures okay (attachment 18)
   leakage currents normal and recorded to spreadsheet (attachment 19)
   good event stats are normal (attachment 20)
   merger running at 3 M items/sec
   tape running at 26 MB/sec

as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space
Attachment 1: 200313_0006_feetemps.png
200313_0006_feetemps.png
Attachment 2: 200313_0008_leakcurrents.png
200313_0008_leakcurrents.png
Attachment 3: 200313_0008_goodevents.png
200313_0008_goodevents.png
Attachment 4: 200313_0015_rateshisto.png
200313_0015_rateshisto.png
Attachment 5: 200313_0018_loweoddfees.png
200313_0018_loweoddfees.png
Attachment 6: 200313_0019_loweevenfees.png
200313_0019_loweevenfees.png
Attachment 7: 200313_0021_higheoddfees.png
200313_0021_higheoddfees.png
Attachment 8: 200313_0022_higheevenfees.png
200313_0022_higheevenfees.png
Attachment 9: 200313_0203_feetemps.png
200313_0203_feetemps.png
Attachment 10: 200313_0203_leakagecurrents.png
200313_0203_leakagecurrents.png
Attachment 11: 200313_0205_goodevents.png
200313_0205_goodevents.png
Attachment 12: 200313_0403_feetemps.png
200313_0403_feetemps.png
Attachment 13: 200313_0408_leakagecurrents.png
200313_0408_leakagecurrents.png
Attachment 14: 200313_0406_goodevents.png
200313_0406_goodevents.png
Attachment 15: 200313_0602_feetemps.png
200313_0602_feetemps.png
Attachment 16: 200313_0605_leakagecurrents.png
200313_0605_leakagecurrents.png
Attachment 17: 200313_0605_goodevents.png
200313_0605_goodevents.png
Attachment 18: 200313_0756_feetemps.png
200313_0756_feetemps.png
Attachment 19: 200313_0758_leakagecurrents.png
200313_0758_leakagecurrents.png
Attachment 20: 200313_0758_goodevents.png
200313_0758_goodevents.png
  442   Fri May 13 07:36:18 2022 OH, TDFriday 13 May
08:36 DAQ Still running smoothly
      Currently on run R1_328
      At current data rates we have just over 4 days of space left on the current disk
      Analysis of R1_327 - attachment 1

14.15 all histograms zero'd
      baseline system wide checks counters
 
      DAQ continues file S450/R4_33

      netint pushenable 6 - was 60 - to investigate FEE64 throughput issues
      netint flushenable 60 - not changed 

      ASIC settings 
       slow comparator aida02 aida04 0x16, aida06 aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      frequency 2Hz
      tau_d 1ms (tail pulse)
      + polarity

Attachments 2 & 3 - ucesb
      Pb beam intensity c. 4e+8/spill - lower than expected 1e+9/spill . Low implant rates (~Hz/spill) are expected for setting.

Attachment 4 & 5 - grafana DSSSD bias, leakage current & temp - OK

Attachments 6-8 - NewMerger stats blocks 0, 1, & 6

Attachment 9 - NewMerger link stats

Attachments 10-11 - TapeServer/NewMerger

Attachment 12 - iptraf aida-gsi network interfaces

Attachments 13-14 - per FEE64 1.8.L spectra
 pulser peak width aida01 123 ch FWHM, aida04 430 ch FWHM

Attachments 15-18 - per FEE64 rates and stat spectra

Attachments 19-22 - system wide checks 
 aida09 clock status 6, WR decoder status errors

Attachments 23-26 - adc, pause, resume correlation sclaer data item stats

Attachment 27 - FEE64 temps OK

Attachment 28 - DSSSSD bias and leakage currents OK

14.30 check ASIC load

14.47 attachment 29 - analysis S450/R4_38

15.11 Attachments 30-35 - data push, flush stats for all FEE64s and aida01 (low ADC data item rate) and aida06 (high ADC data item rate)
Attachment 1: 220513_0839_R1_327_analysis.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259232212 ( 1719602.4 Hz)
        Other data format:    2687788 (   17829.3 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        986 (       6.5 Hz)
                               RESUME:        986 (       6.5 Hz)
                              SYNC100:      32562 (     216.0 Hz)
                              WR48-63:      32562 (     216.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2620692 (   17384.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     130683 (     866.9 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      150.751 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.085                0.000
  1                9.978                0.000
  2                0.059                0.000
  3               21.640                0.000
  4                0.158                0.000
  5               66.897                0.000
  6                1.778                0.000
  7               36.972                0.000
  8                0.172                0.000
  9                0.018                0.000
 10                0.000                7.315
 11               28.135                0.000
 12                0.007                0.000
 13                0.000                3.034
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    7933827       2887          0          0          5          5        987        987          0        903          0       1141
  1   37526918       9508          0          0         50         50       4704       4704          0          0          0     122502
  2    8368319    1400718          0          0          2          2       1222       1222          0    1398270          0        647
  3   39185673    1207805          0          0        156        156       5116       5116          0    1197261          0        703
  4    4609566       1178          0          0          5          5        584        584          0          0          0       1170
  5   40049357      10540          0          0        309        309       4961       4961          0          0          0        297
  6   13162553      17502          0          0         13         13       1685       1685          0      14106          0        302
  7   41199857      20850          0          0        248        248       5101       5101          0      10152          0        241
  8   12021437       2864          0          0          5          5       1427       1427          0          0          0        467
  9    4748197       1188          0          0          2          2        592        592          0          0          0       1165
 10    3089949        744          0          0          0          0        372        372          0          0          0        247
 11   38258323       9856          0          0        190        190       4738       4738          0          0          0       1052
 12    6307578       1518          0          0          1          1        758        758          0          0          0        311
 13    2770658        630          0          0          0          0        315        315          0          0          0        438
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   33.600s (  952.392 blocks/s,  59.525 Mb/s)
Attachment 2: Screenshot_from_2022-05-13_14-28-00.png
Screenshot_from_2022-05-13_14-28-00.png
Attachment 3: Screenshot_from_2022-05-13_14-27-51.png
Screenshot_from_2022-05-13_14-27-51.png
Attachment 4: Screenshot_from_2022-05-13_14-27-31.png
Screenshot_from_2022-05-13_14-27-31.png
Attachment 5: Screenshot_from_2022-05-13_14-27-17.png
Screenshot_from_2022-05-13_14-27-17.png
Attachment 6: Screenshot_from_2022-05-13_14-26-32.png
Screenshot_from_2022-05-13_14-26-32.png
Attachment 7: Screenshot_from_2022-05-13_14-26-26.png
Screenshot_from_2022-05-13_14-26-26.png
Attachment 8: Screenshot_from_2022-05-13_14-26-21.png
Screenshot_from_2022-05-13_14-26-21.png
Attachment 9: Screenshot_from_2022-05-13_14-26-08.png
Screenshot_from_2022-05-13_14-26-08.png
Attachment 10: Screenshot_from_2022-05-13_14-25-46.png
Screenshot_from_2022-05-13_14-25-46.png
Attachment 11: Screenshot_from_2022-05-13_14-25-38.png
Screenshot_from_2022-05-13_14-25-38.png
Attachment 12: Screenshot_from_2022-05-13_14-25-22.png
Screenshot_from_2022-05-13_14-25-22.png
Attachment 13: Screenshot_from_2022-05-13_14-24-07.png
Screenshot_from_2022-05-13_14-24-07.png
Attachment 14: Screenshot_from_2022-05-13_14-22-44.png
Screenshot_from_2022-05-13_14-22-44.png
Attachment 15: Screenshot_from_2022-05-13_14-21-16.png
Screenshot_from_2022-05-13_14-21-16.png
Attachment 16: Screenshot_from_2022-05-13_14-21-04.png
Screenshot_from_2022-05-13_14-21-04.png
Attachment 17: Screenshot_from_2022-05-13_14-20-34.png
Screenshot_from_2022-05-13_14-20-34.png
Attachment 18: Screenshot_from_2022-05-13_14-20-14.png
Screenshot_from_2022-05-13_14-20-14.png
Attachment 19: Screenshot_from_2022-05-13_14-19-10.png
Screenshot_from_2022-05-13_14-19-10.png
Attachment 20: Screenshot_from_2022-05-13_14-18-40.png
Screenshot_from_2022-05-13_14-18-40.png
Attachment 21: Screenshot_from_2022-05-13_14-18-31.png
Screenshot_from_2022-05-13_14-18-31.png
Attachment 22: Screenshot_from_2022-05-13_14-18-14.png
Screenshot_from_2022-05-13_14-18-14.png
Attachment 23: Screenshot_from_2022-05-13_14-17-38.png
Screenshot_from_2022-05-13_14-17-38.png
Attachment 24: Screenshot_from_2022-05-13_14-17-23.png
Screenshot_from_2022-05-13_14-17-23.png
Attachment 25: Screenshot_from_2022-05-13_14-17-09.png
Screenshot_from_2022-05-13_14-17-09.png
Attachment 26: Screenshot_from_2022-05-13_14-16-56.png
Screenshot_from_2022-05-13_14-16-56.png
Attachment 27: Screenshot_from_2022-05-13_14-16-30.png
Screenshot_from_2022-05-13_14-16-30.png
Attachment 28: Screenshot_from_2022-05-13_14-16-11.png
Screenshot_from_2022-05-13_14-16-11.png
Attachment 29: R4_38.txt
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259102731 ( 1572653.5 Hz)
        Other data format:    2817271 (   17099.7 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:       2412 (      14.6 Hz)
                               RESUME:       2414 (      14.7 Hz)
                              SYNC100:      32614 (     198.0 Hz)
                              WR48-63:      32614 (     198.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2747217 (   16674.5 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:      30594 (     185.7 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      164.755 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                5.297                0.000
  2                0.000                0.000
  3               18.287                0.000
  4                0.000                0.000
  5               77.894                0.000
  6                0.000                0.000
  7               25.402                0.000
  8                0.001                0.000
  9                0.000                0.000
 10                0.000                2.705
 11               15.617                0.000
 12                0.000                0.000
 13                0.000               29.550
 14                0.000                0.000
 15                0.000                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    6844719       2775          0          0          0          0        894        894          0        987          0        375
  1   39920437      10378          0          0        277        277       4912       4912          0          0          0      27546
  2    7063505    1453891          0          0          0          0       1085       1085          0    1451721          0        328
  3   42633132    1301928          0          0        410        410       5590       5590          0    1289928          0        251
  4    3625899        900          0          0          0          0        450        450          0          0          0        125
  5   38867835      10853          0          0        768        769       4658       4658          0          0          0        129
  6    7590830       4334          0          0          0          0        934        934          0       2466          0        227
  7   44498336      14120          0          0        564        565       5438       5438          0       2115          0        113
  8   10949373       2716          0          0          1          1       1357       1357          0          0          0        345
  9    3887105        996          0          0          0          0        498        498          0          0          0        273
 10    1999345        532          0          0          0          0        266        266          0          0          0        160
 11   43113077      11754          0          0        392        392       5485       5485          0          0          0        288
 12    6030564       1596          0          0          0          0        798        798          0          0          0        252
 13    2078574        498          0          0          0          0        249        249          0          0          0        182
 14          0          0          0          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   32.668s (  979.553 blocks/s,  61.222 Mb/s)
Attachment 30: Screenshot_from_2022-05-13_15-11-36.png
Screenshot_from_2022-05-13_15-11-36.png
Attachment 31: Screenshot_from_2022-05-13_15-11-29.png
Screenshot_from_2022-05-13_15-11-29.png
Attachment 32: Screenshot_from_2022-05-13_15-11-17.png
Screenshot_from_2022-05-13_15-11-17.png
Attachment 33: Screenshot_from_2022-05-13_15-10-55.png
Screenshot_from_2022-05-13_15-10-55.png
Attachment 34: Screenshot_from_2022-05-13_15-10-41.png
Screenshot_from_2022-05-13_15-10-41.png
Attachment 35: Screenshot_from_2022-05-13_15-10-30.png
Screenshot_from_2022-05-13_15-10-30.png
  704   Fri Jun 13 12:42:39 2025 CC, TDFriday 13 June 2025
13.42 ASIC settings 2025Jun12-13.53.04
      *all* FEE64s slow comparator 0x64
                   all fast disc outputs disabled
      p+n FEE64s waveform threshold 7000, n+n FEE64s 9000

      CC grounding ribbon cable heavy duty braid to adaptor PCB and AIDA support assembly ground
       'top' p+n FEE64s and snout to be completed
      
16.00 DSSSD bias & leakage current - attachment 1
      FEE64 temps OK - attachment 2
      All system wide checks OK *except* WR decoder status aida02 and aida06 - attachment 3
      WR timestamps OK - attachment 4
      ADC data item stats - attachment 5
      per FEE64 Rate spectra - attachment 6
      per p+n FEE64 1.8.L spectra - attachment 7
       aida09 pulser peak width 117 ch FWHM - no change since yesterday
      per FEE64 1.8.W spectra - 20us FSR - attachments 8-10
  
Attachment 1: Screenshot_from_2025-06-13_15-48-17.png
Screenshot_from_2025-06-13_15-48-17.png
Attachment 2: Screenshot_from_2025-06-13_15-48-34.png
Screenshot_from_2025-06-13_15-48-34.png
Attachment 3: Screenshot_from_2025-06-13_15-48-56.png
Screenshot_from_2025-06-13_15-48-56.png
Attachment 4: Screenshot_from_2025-06-13_15-49-36.png
Screenshot_from_2025-06-13_15-49-36.png
Attachment 5: Screenshot_from_2025-06-13_15-55-42.png
Screenshot_from_2025-06-13_15-55-42.png
Attachment 6: Screenshot_from_2025-06-13_15-56-55.png
Screenshot_from_2025-06-13_15-56-55.png
Attachment 7: Screenshot_from_2025-06-13_15-58-10.png
Screenshot_from_2025-06-13_15-58-10.png
Attachment 8: Screenshot_from_2025-06-13_15-58-48.png
Screenshot_from_2025-06-13_15-58-48.png
Attachment 9: Screenshot_from_2025-06-13_15-59-12.png
Screenshot_from_2025-06-13_15-59-12.png
Attachment 10: Screenshot_from_2025-06-13_16-00-09.png
Screenshot_from_2025-06-13_16-00-09.png
  192   Fri Mar 12 10:09:19 2021 LS, CAFriday 12th March 11.00-
11.00(Germany) System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment1)
      Spectra rates (attachment2)
      FEE temps (attachment3)
      Leakage currents, written to google sheet (attachment4)  
      Merger~ 4.9M items/s
      Tapeserver ~17MB/s 

      In MBS control terminal, connection has been closed intentionally since this morning (file S452f160),
      AIDA has been taken out of the timesorter due to the high data rate, buffers were full
      AIDA cannot be seen in ucesb or Go4.

13.00 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment5)
      Spectra rates (attachment6)
      FEE temps (attachment7)
      Leakage currents, written to google sheet (attachment8)  
      Merger~ 5.1M items/s
      Tapeserver ~18MB/s 

      No timestamp related errors this shift

13.37 AIDA MBS control restarted R33_388
      System wide checks same as previous time

13.54 beam stopped for access,  file R33_396

13.57 seen recent batch of bad timestamp errors in new merger terminal (attachment9) should be around R33_397
      Analysed R33_395, 396, 397, 398 no timewarps
      
14.29 beam back R33_415

14.54 see more bad timestamps in new merger terminal (attachment10)

14.55 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment11)
      Spectra rates (attachment12)
      FEE temps (attachment13)
      Leakage currents, written to google sheet (attachment14)  
      Merger~ 5.0M items/s
      Tapeserver ~17MB/s 

16.05 AIDA included back into timesorter, AIDA scalers now seen in ucesb, file R33_463

16.15 CA takes over until 18:00

17:11 System wide checks:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0xcdd6 : 	 0xcdd7 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

17:13 FEE64 temps ok - attachment 15
      Statistics ok - attachment 16
      bias and leakage currents ok - attachment 17
Attachment 1: 1100_Statistics.PNG
1100_Statistics.PNG
Attachment 2: 1100_SpectraRate.PNG
1100_SpectraRate.PNG
Attachment 3: 1100_FEETemps.PNG
1100_FEETemps.PNG
Attachment 4: 1100_LeakageCurrents.PNG
1100_LeakageCurrents.PNG
Attachment 5: 1300_Statistics.PNG
1300_Statistics.PNG
Attachment 6: 1300_SpectraRate.PNG
1300_SpectraRate.PNG
Attachment 7: 1300_FEETemps.PNG
1300_FEETemps.PNG
Attachment 8: 1300_LeakageCurrents.PNG
1300_LeakageCurrents.PNG
Attachment 9: 1358_NewMerger.PNG
1358_NewMerger.PNG
Attachment 10: 1454_NewMerger.PNG
1454_NewMerger.PNG
Attachment 11: 1500_Statistics.PNG
1500_Statistics.PNG
Attachment 12: 1500_SpectraRate.PNG
1500_SpectraRate.PNG
Attachment 13: 1500_FEETemps.PNG
1500_FEETemps.PNG
Attachment 14: 1500_LeakageCurrents.PNG
1500_LeakageCurrents.PNG
Attachment 15: Screenshot_from_2021-03-12_16-18-13.png
Screenshot_from_2021-03-12_16-18-13.png
Attachment 16: Screenshot_from_2021-03-12_16-18-51.png
Screenshot_from_2021-03-12_16-18-51.png
Attachment 17: Screenshot_from_2021-03-12_16-21-21.png
Screenshot_from_2021-03-12_16-21-21.png
  191   Fri Mar 12 08:32:46 2021 TDFriday 12 March 08.00-
08.31 DAQ continues OK - file R33_263

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

08.32 No merger server error/warning messages since most recent restart c. 00.00 today


08.44 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x7a56 : 	 0x7a58 : 	 2  
aida06 fault 	 0x65bb : 	 0x65bd : 	 2  
aida07 fault 	 0xcdd1 : 	 0xcdd4 : 	 3  
aida08 fault 	 0x2ab3 : 	 0x2ab5 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     14      6      3      2      1      2      2      3      3      4     11   : 58904
aida02 :      6      4      1      1      1      2      2      4      2      3      7   : 39848
aida03 :      5      6      4      2      1      3      1      3      2      3      7   : 39300
aida04 :      4      1      2      2      3      3      2      4      1      4     15   : 73912
aida05 :      1      5      2      2      2      4      2      3      2      3      7   : 39692
aida06 :      7      2      2      3      1      3      3      3      2      4      7   : 41836
aida07 :      6      3      1      2      3      2      2      3      3      3      7   : 40512
aida08 :      8      5      2      2      1      3      1      3      2      3      7   : 39272
aida09 :      1      2      3      1      1      3      2      3      1      4      7   : 40484
aida10 :      2      6      3      2      1      2      2      2      3      3      7   : 39912
aida11 :      2      3      1      0      2      4      1      3      2      3      7   : 39344
aida12 :      2      3      4      2      0      3      3      3      2      3      7   : 39712


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 5.1M data items/s
      TapeServer OK - 16Mb/s

08.54 p+n junction strip HEC spectra - attachment 1
      Rate spectra - attachment 2

09.15 Burst of Merger server warning messages of type

MERGE Data Link (3547): bad timestamp  1 3 0x8128d6c8 0x05056328 0x00008d6c85056328 0x166b8d6c85056328 0x166b8d6d86da01ee

      also multiple

Warning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one 
MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in relayWarning: At least one MIDAS block missed in 
relayWarning: At least one MIDAS block missed in relayWarning: A

      The latter may reflect downstream problems writing MBS data
Attachment 1: Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-12_Statistics_aidas-gsi.png
Screenshot_2021-03-12_Statistics_aidas-gsi.png
Attachment 3: 30.png
30.png
Attachment 4: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
Attachment 5: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi.png
  193   Fri Mar 12 20:12:10 2021 TDFriday 12 March
21.12 DAQ continues OK - file R33_617

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

21.15 System wide checks


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x7a58 : 	 0x7a5a : 	 2  
aida06 fault 	 0x65bd : 	 0x65bf : 	 2  
aida07 fault 	 0xcdd6 : 	 0xcdda : 	 4  
aida08 fault 	 0x2ab5 : 	 0x2ab7 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      4      2      0      1      2      3      3      2      3     11   : 55956
aida02 :     21      4      1      2      2      4      2      2      1      4      7   : 40260
aida03 :     10      4      4      1      6      4      1      3      3      3      6   : 36648
aida04 :      7      2      4      0      2      3      2      4      1      4     15   : 73836
aida05 :      1      3      1      5      4      3      1      2      3      4      6   : 37964
aida06 :     12      3      5      2      1      3      3      3      2      4      7   : 41880
aida07 :      2      4      3      3      4      2      2      4      3      3      6   : 37048
aida08 :     17     13      2      3      2      3      2      3      2      3      7   : 39724
aida09 :      3      7      2      0      1      2      2      3      2      2      7   : 37284
aida10 :      2      5      3      2      2      3      1      3      2      3      7   : 39328
aida11 :     23     18      2      3      2      3      3      2      3      3      6   : 36460
aida12 :     18      7      3      1      1      4      2      2      2      3      7   : 39184

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.9M data items/s
      TapeServer OK - 16Mb/s

      All histograms zero'd

21.50 At some point between R33_610 (20.59) and R33_625 (21.30) aida05 stopped producing data (zero good events - see attachment 2)

      able to telnet to aida05 - no warnings/error messages in /var/log/messages

      DAQ STOP (all except aida05 stopped OK, aida05 remained GOING)

      issued aida05 reboot command via telnet command line

      DAQ RESET/SETUP/GO (all FEE64s GOING OK except aida05 - zero good events)

21.15 All system wide checks OK *except*

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


21.50 aida04 & aida05 statistics for comparison - see attachments 4 & 5
Attachment 1: Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-12_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 2: Screenshot_2021-03-12_Statistics_aidas-gsi(1).png
Screenshot_2021-03-12_Statistics_aidas-gsi(1).png
Attachment 3: 31.png
31.png
Attachment 4: Screenshot_2021-03-12_Statistics_aidas-gsi(2).png
Screenshot_2021-03-12_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-03-12_Statistics_aidas-gsi(3).png
Screenshot_2021-03-12_Statistics_aidas-gsi(3).png
  395   Fri Feb 11 12:09:24 2022 NHFri 11 Feb AIDA Work
Use beam break between FRS experiments to finalise getting AIDA testable for PJCS etc.

USB hubs reconnected - all 16 ttyUSBs seem to be visible now

New inverter installed (Ortec 533 Sum Invert, brand new) - Pulser T connected to input B1 and inverted out B sent to -ve FEEs

VME crate for VETAR2 is powered on so White Rabbit should be available

All grounds reconnected on FEEs

All FEEs connected to pulser chain again

-

Moved interlock to new Al shelf on AIDA rack so it's not kicked
Powered on OK, all 4 lights on in box but AIDA Relay had to water light
Investigation shows the cable between relay and interlock was broken, one wire had come loose from connector
Resoldered and now it works as usual

System powered up OK
All 16 FEEs respond to serial
All 16 FEEs have WR
All 16 FEEs show pulser ADC entries

Seems good!
  479   Fri Jun 10 10:32:46 2022 NHFri 10 Jun
Probe PSU output with 'scope again and 10X probe

Use a AC mains scope as was available, means referenced to ground of AC, but should be ground of platform (and hence Fee64)

Figs 1-3: Traces of PSU outputs (5V, -6V, 5V Return) with 10X probe, 1 MOhm impedance AC coupled 
          Lots of fluctuation and jitter

Figs 4-6: DC 1MOhm output from all 3 power rails 

Fig 7-8: Small "loop" using probe and observed scope spectra.

After lunch ideas:

- Try again with 1X probe attenuation, maybe record output to USB 

- Remove unconnected MACBs from master

- 
Attachment 1: scopePSU1.jpg
scopePSU1.jpg
Attachment 2: scopePSU2.jpg
scopePSU2.jpg
Attachment 3: scoepPSU3.jpg
scoepPSU3.jpg
Attachment 4: scope5V_DC.jpg
scope5V_DC.jpg
Attachment 5: scopeN6V_DC.jpg
scopeN6V_DC.jpg
Attachment 6: scope_7VDC.jpg
scope_7VDC.jpg
Attachment 7: scope_RFAnt.jpg
scope_RFAnt.jpg
Attachment 8: scopeRF_pickup.jpg
scopeRF_pickup.jpg
ELOG V3.1.3-7933898