AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40 MONNET
  DESPEC, Page 20 of 37  ELOG logo
ID Date Author Subjectdown
  372   Mon Jun 21 10:21:50 2021 NHMonday 21 June
11.21 - DAQ continues OK (no storage mode)

11.21 - System checks ok *except*

		 Base 		Current 	Difference
aida01 fault 	 0x8944 : 	 0x8963 : 	 31  
aida02 fault 	 0x9b36 : 	 0x9b55 : 	 31  
aida03 fault 	 0xf26b : 	 0xf28a : 	 31  
aida04 fault 	 0x57a0 : 	 0x57bf : 	 31  
aida05 fault 	 0x4d3c : 	 0x4d95 : 	 89  
aida05 : WR status 0x10
 aida06 fault 	 0x640c : 	 0x6425 : 	 25  
aida07 fault 	 0x255e : 	 0x2577 : 	 25  
aida08 fault 	 0xffbd : 	 0xffd6 : 	 25  
aida09 fault 	 0x87e : 	 0x895 : 	 23  
aida10 fault 	 0xc557 : 	 0xc56e : 	 23  
aida11 fault 	 0x683e : 	 0x6855 : 	 23  
aida12 fault 	 0x1c35 : 	 0x1c4c : 	 23  
aida13 fault 	 0x8f41 : 	 0x8f58 : 	 23  
aida14 fault 	 0x1b47 : 	 0x1b5e : 	 23  
aida15 fault 	 0x4e4 : 	 0x127e : 	 3482  
aida16 fault 	 0xf570 : 	 0xf587 : 	 23  
White Rabbit error counter test result: Passed 0, Failed 16


			 Base 		Current 		Difference
aida01 fault 	 0x0 : 	 0x2 : 	 2  
aida02 fault 	 0x0 : 	 0x2 : 	 2  
aida05 fault 	 0x0 : 	 0x3 : 	 3  
aida06 fault 	 0x0 : 	 0x2 : 	 2  
aida09 fault 	 0x0 : 	 0x2 : 	 2  
aida10 fault 	 0x0 : 	 0x2 : 	 2  
aida11 fault 	 0x0 : 	 0x2 : 	 2  
aida12 fault 	 0x0 : 	 0x9560 : 	 38240  
aida13 fault 	 0x0 : 	 0x15b9 : 	 5561  
aida14 fault 	 0x0 : 	 0x2 : 	 2  
aida15 fault 	 0x0 : 	 0x2 : 	 2  
aida16 fault 	 0x0 : 	 0x2 : 	 2  
FPGA Timestamp error counter test result: Passed 4, Failed 12
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     33     12     14      2      4      4      2      2      2      4      6   : 37636
aida02 :     40      7     15      1      3      4      3      2      2      4      6   : 37800
aida03 :     39      7      9      6      5      3      1      3      2      4      6   : 37860
aida04 :     27      7      5      3      3      4      2      3      2      4      6   : 37908
aida05 :     33      5      5      3      6      4      3      3      2      4      6   : 38364
aida06 :     45     12      3      4      4      2      2      3      4      3      6   : 37828
aida07 :     31     11      5      4      4      2      3      2      2      4      6   : 37540
aida08 :     31      6      8      5      3      4      3      2      2      4      6   : 37772
aida09 :     45      4      9      3      2      3      3      2      2      4      6   : 37572
aida10 :     37     11     10      2      3      4      3      3      3      3      6   : 37260
aida11 :     40      9     11      3      3      4      3      3      1      4      6   : 37304
aida12 :     44     11      6      5      3      4      3      3      1      4      6   : 37320
aida13 :     41     11      9      3      5      3      3      3      1      4      6   : 37292
aida14 :     31     12      4      6      5      3      3      3      3      3      6   : 37276
aida15 :     17      9      7      1      2      2      3      2      1      4      7   : 40348
aida16 :     41     12      5      4      5      3      3      3      1      4      6   : 37268


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sat Jun 12 15:46:33 CEST 2021
 FEE : aida02 =>   Options file size is 1025 	 Last changed Sun May 23 00:19:21 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Fri May 14 16:54:56 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Mon May 17 06:25:41 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Sun May 23 00:16:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1025 	 Last changed Sat Jun 12 15:48:28 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021


11.23 - figures
    Grafana temp & leakage - attachment 1
    Weather cooler this week in Germany, expect S4 to cool down eventually
    Lost activity monitor - attachment 2
    1.8.W spectra - 20us FSR - attachments 3 & 4
    aida05 looks very noisy (?) 
    ADC data items - attachment 5
    FEE64 temperatures OK - attachment 6
    DSSSD bias & leakage currents - attachment 7
    Merger/Tape Server/Merger statistics - attachment 8
    Looks some new time seq error since Thursday

Experiment is runnign in S4 so no access possible this week

Data is being archived to lustre (GSI PB file storage) 
Attachment 1: firefox_2021-06-21_11-27-00.png
firefox_2021-06-21_11-27-00.png
Attachment 2: AnyDesk_2021-06-21_11-24-28.png
AnyDesk_2021-06-21_11-24-28.png
Attachment 3: AnyDesk_2021-06-21_11-25-44.png
AnyDesk_2021-06-21_11-25-44.png
Attachment 4: AnyDesk_2021-06-21_11-26-07.png
AnyDesk_2021-06-21_11-26-07.png
Attachment 5: AnyDesk_2021-06-21_11-24-15.png
AnyDesk_2021-06-21_11-24-15.png
Attachment 6: AnyDesk_2021-06-21_11-24-43.png
AnyDesk_2021-06-21_11-24-43.png
Attachment 7: AnyDesk_2021-06-21_11-25-04.png
AnyDesk_2021-06-21_11-25-04.png
Attachment 8: AnyDesk_2021-06-21_11-25-15.png
AnyDesk_2021-06-21_11-25-15.png
  486   Mon Jun 20 09:26:23 2022 OH, TDMonday 20 June
10:26 MACB HDMI for AIDA03 and AIDA04 reseated.
      Merger and DAQ restarted - All happily sending data
      Has been running for 10 minutes with no WR errors.
      Rates from yesterday achieved again.

      DTAS will move back into position at some point.
      Once this is done, will perform pulser walkthrough.


11:14 DTAS now back in position for experiment
      Slight increase in noise noticed across a few FEEs
      Over time this has decreased on all but AIDA07 which is now running at ~80k - attachment 1
      AIDA03 is now however gaining white rabbit errors at a rate of 1 every few minutes.
      Less frequently gaining FPGA errors
    
12:20 AIDA power cycle to reseat the HDMI for AIDA03 - It would eventually crash if not done
      Upon powering up again rates in AIDA08 slightly worse - attachment 2

13:26 No WR errors since the last power cycle.

21:16 Baseline system wide checks, zero histograms & stats

All system wide checks OK

per p+n FEE64 1.8.L spectra - attachment 3
 aida01 pulser peak width 82 channels FWHM

per FEE64 rate spectra - attachment 4

adc, pause, resume and correlation scaler data item stats - attachments 5-8

FEE64 temps OK - attachment 9

DSSSD bias & leakage currents OK - attachments 10 & 11
Attachment 1: 220620_1116_stats_no_waveform.png
220620_1116_stats_no_waveform.png
Attachment 2: 220620_1223_stats_no_waveform.png
220620_1223_stats_no_waveform.png
Attachment 3: Screenshot_from_2022-06-20_21-17-41.png
Screenshot_from_2022-06-20_21-17-41.png
Attachment 4: Screenshot_from_2022-06-20_21-14-12.png
Screenshot_from_2022-06-20_21-14-12.png
Attachment 5: Screenshot_from_2022-06-20_21-12-06.png
Screenshot_from_2022-06-20_21-12-06.png
Attachment 6: Screenshot_from_2022-06-20_21-10-55.png
Screenshot_from_2022-06-20_21-10-55.png
Attachment 7: Screenshot_from_2022-06-20_21-08-58.png
Screenshot_from_2022-06-20_21-08-58.png
Attachment 8: Screenshot_from_2022-06-20_21-04-58.png
Screenshot_from_2022-06-20_21-04-58.png
Attachment 9: Screenshot_from_2022-06-20_21-04-00.png
Screenshot_from_2022-06-20_21-04-00.png
Attachment 10: Screenshot_from_2022-06-20_21-02-07.png
Screenshot_from_2022-06-20_21-02-07.png
Attachment 11: Screenshot_from_2022-06-20_21-01-07.png
Screenshot_from_2022-06-20_21-01-07.png
  248   Mon Apr 19 16:58:29 2021 LJWMonday 19th April 16:00-20:00 Shift
System Checks @ ~18:10 :


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	
		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd05a : 	 8  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x23 : 	 35  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      6      4      0      1      5      2      3      2      3      6   : 35636
aida02 :     27      3      2      0      2      3      2      3      3      3      6   : 36516
aida03 :     33      9      2      1      1      3      2      3      3      3      6   : 36556
aida04 :     11      6      5      4      2      4      3      3      3      3      6   : 37036
aida05 :     15      8      7      4      3      4      4      2      3      3      6   : 36908
aida06 :     13      9      7      3      3      4      2      3      3      3      6   : 36876
aida07 :      1      3      3      2      2      3      1      3      3      3      6   : 36236
aida08 :     29      5      2      1      1      2      2      3      3      3      6   : 36380
aida09 :      3      8      2      0      2      3      2      2      2      4      6   : 36972
aida10 :      9      5      3      3      1      4      2      4      2      3      6   : 36124
aida11 :      5      7      3      2      2      2      2      4      2      3      6   : 35900
aida12 :     24      6      3      1      2      3      2      2      3      3      6   : 36064


System Checks @ ~19:55

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd05e : 	 12  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x27 : 	 39  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     18     10      2      2      1      5      2      3      2      3      6   : 35768
aida02 :     21      9      2      1      2      2      2      3      3      3      6   : 36444
aida03 :     13      5      6      1      2      2      2      3      3      3      6   : 36444
aida04 :     30     11      4      3      2      4      3      3      3      3      6   : 37104
aida05 :     10      9      8      4      4      4      4      2      3      3      6   : 36976
aida06 :     22     17     10      3      2      3      1      4      3      3      6   : 37088
aida07 :     20      8      4      1      2      4      1      2      3      3      6   : 35952
aida08 :     14      4      3      0      1      2      2      3      3      3      6   : 36296
aida09 :     24      8      4      0      1      3      3      3      3      3      6   : 36768
aida10 :     19      8      4      1      1      4      2      3      3      3      6   : 36652
aida11 :      6      6      1      2      3      1      2      4      2      3      6   : 35800
aida12 :     15      7      1      1      2      4      1      2      3      3      6   : 35876
Attachment 1: 63.png
63.png
Attachment 2: Screenshot_2021-04-19_Statistics_aidas-gsi(3).png
Screenshot_2021-04-19_Statistics_aidas-gsi(3).png
Attachment 3: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(4).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(4).png
Attachment 4: 64.png
64.png
Attachment 5: 65.png
65.png
Attachment 6: 66.png
66.png
  246   Mon Apr 19 13:27:08 2021 RDPMonday 19th April 12 noon shift
12.30 There appear to be fewer HEC events in aida09 cf. aida11 and upstream DSSSDs
      Decrease aida09 asic #1-4 HEC fast comparator 0x2 -> 0x1
      Online/nearline analyses of daat files indicates that this does not resolve the issue

Some timestamp issues 

MERGE Data Link (23725): bad timestamp  11 3 0x8b620008 0x0287be94 0x00003dd78287be94 0x16773dd78287be94 0x16773dd78287beb2
MERGE Data Link (23725): bad timestamp  11 3 0x8b610040 0x073302ce 0x00003df3e73302ce 0x16773df3e73302ce 0x16773df3e73302ec
MERGE Data Link (23725): bad timestamp  11 3 0x8b610020 0x0a54b794 0x00003e4e4a54b794 0x16773e4e4a54b794 0x16773e4e4a54b7b2
MERGE Data Link (23725): bad timestamp  11 3 0x8b610020 0x052da21e 0x00003f68f52da21e 0x16773f68f52da21e 0x16773f68f52da232
MERGE Data Link (23725): bad timestamp  11 3 0x8b620008 0x0061489e 0x000040aa8061489e 0x167740aa8061489e 0x167740aa806148a8
MERGE Data Link (23725): bad timestamp  11 3 0x8b610008 0x04029d5e 0x00004169c4029d5e 0x16774169c4029d5e 0x16774169c4029d68
MERGE Data Link (23725): bad timestamp  11 3 0x8b630004 0x08dfcb14 0x0000417fd8dfcb14 0x1677417fd8dfcb14 0x1677417fd8dfcb28
MERGE Data Link (23725): bad timestamp  11 3 0x8b630020 0x08dfcb1e 0x0000417fd8dfcb1e 0x1677417fd8dfcb1e 0x1677417fd8dfcb28

13.50 Increased DSSSD# 3 bias from -100V

      -100V -6.325uA
      -110V -6.500uA
      -120V -6.800uA



       
      
Attachment 1: 61.png
61.png
Attachment 2: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(2).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(2).png
Attachment 3: Screenshot_2021-04-19_Statistics_aidas-gsi(2).png
Screenshot_2021-04-19_Statistics_aidas-gsi(2).png
  243   Mon Apr 19 01:08:40 2021 MAMonday 19th April 00:00-08:00

03:09   General check

Rates, temptature, voltages are OK attached 1, 2, 3, 4

 

****Clock Ckeck******

OK
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

******ADC check ******
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

****** White Rabbit check******

     Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c2c :      48  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

*****FPGA check ******


             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x10 :      6  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

*****Memory check*****


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      6     12     11      7      2      3      2      4      2      3      6   : 36360
aida02 :     21     15      7      3      2      3      1      4      2      3      6   : 35996
aida03 :      9     10     10      2      4      3      3      3      2      3      6   : 36052
aida04 :     17     17     17      4      0      2      3      4      2      3      6   : 36444
aida05 :     25      7      3      1      1      1      2      3      2      4      6   : 37292
aida06 :     29     15      7      2      3      4      1      3      3      3      6   : 36700
aida07 :     17     17      6      5      2      3      2      3      3      3      6   : 36812
aida08 :     22      6     10      5      3      1      3      4      2      3      6   : 36360
aida09 :     18     28     14      4      2      3      3      3      2      3      6   : 36232
aida10 :     26     16      4      3      3      1      3      4      2      3      6   : 36296
aida11 :     16     10      4      0      3      3      1      4      2      3      6   : 35856
aida12 :     19     14      4      2      4      4      1      3      3      3      6   : 36668

 

04:15      General Check

Rates, Tempratures, Voltages are ok, attached 5,6,7,8

******Clock Check*******

OK

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

******ADC******

Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

******White Rabbit check******


         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c39 :      61  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


******FPGA Check******

     Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x11 :      7  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


****** Memorey check******


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     27     21     10      7      1      3      2      4      2      3      6   : 36436
aida02 :     28     21      3      3      1      3      1      4      2      3      6   : 35944
aida03 :     15      8     11      3      3      3      3      3      2      3      6   : 36044
aida04 :     25     19     14      4      1      3      4      3      2      3      6   : 36380
aida05 :     12      6      5      1      0      2      2      3      2      4      6   : 37328
aida06 :     29     21      4      2      3      4      1      3      3      3      6   : 36700
aida07 :     19     14      7      6      2      3      2      2      3      3      6   : 36332
aida08 :     11      8      8      6      3      1      3      4      2      3      6   : 36332
aida09 :     16     25     11      3      1      2      2      4      2      3      6   : 36184
aida10 :     14     13      4      2      3      2      2      4      2      3      6   : 36064
aida11 :      4      9      4      0      3      3      1      4      2      3      6   : 35800
aida12 :     38     17      4      2      4      4      1      3      3      3      6   : 36768

 

 

 

 

Attachment 1: 210419_0302_Voltages.png
210419_0302_Voltages.png
Attachment 2: 210419_0303_Stats.png
210419_0303_Stats.png
Attachment 3: 210419_0303_Temp.png
210419_0303_Temp.png
Attachment 4: 210419_0406_ucesb.png
210419_0406_ucesb.png
Attachment 5: 210419_0402_Stats.png
210419_0402_Stats.png
Attachment 6: 210419_0402_Voltages.png
210419_0402_Voltages.png
Attachment 7: 210419_0406_ucesb.png
210419_0406_ucesb.png
Attachment 8: 210419_0414_ucesb.png
210419_0414_ucesb.png
  249   Mon Apr 19 19:19:18 2021 CBMonday 19 April 20:00-24:00 shift
20:19 
Statistics OK (attach 1)
No faults found
Ucesb OK (attach 2)
Temps OK (attach 3)
Bias OK (attach 4)

System-wide checks - as per last check
aida09 fails clock (OK)
aida07 and 09 fail calibration (OK)

		 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd05f : 	 13  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x29 : 	 41  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      0      3      2      2      2      5      2      3      2      3      6   : 35704
aida02 :     19      4      1      1      1      2      2      3      3      3      6   : 36316
aida03 :     12      9      2      1      2      2      2      3      3      3      6   : 36408
aida04 :     16      3      5      4      2      4      2      3      3      3      6   : 36776
aida05 :     41     10     10      3      3      3      3      3      3      3      6   : 37172
aida06 :     25     21      9      3      2      3      1      4      3      3      6   : 37116
aida07 :     10      7      1      2      1      3      2      2      3      3      6   : 35952
aida08 :     33      7      5      1      1      2      2      3      3      3      6   : 36460
aida09 :     12      7      4      2      2      2      2      2      2      4      6   : 36968
aida10 :     16      6      2      1      1      4      3      3      2      3      6   : 35824
aida11 :      2      0      1      2      2      2      2      4      2      3      6   : 35800
aida12 :     15      6      2      0      2      4      2      2      3      3      6   : 36108



20:48 Started compressing *old* runs on Screen 2, Workspace 5
nice -n10 gzip -v R[...]_*

21:00 Stats OK, UCESB OK, No faults

21:07 Go4 spectra erased by remote shifters.

21:15 Local shifters report correlations lost. Beam may be intermittently removed for checks.

21:30 Still no beam. DAQ issues locally. AIDA MBS link had to be restarted due to operator mistake.




21:37 Beam back. Stats OK, UCESB OK, No faults found.

System wide checks:
Clock - aida09 fails

All modules now fail ADC calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd061 : 	 15  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

	
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x36 : 	 54  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      3      1      2      4      2      4      2      3      6   : 36104
aida02 :      2      3      0      2      2      4      2      2      3      3      6   : 36064
aida03 :      6      3      3      3      2      2      2      2      4      3      6   : 36928
aida04 :     15      5      8      4      3      3      2      3      3      3      6   : 36772
aida05 :     10      5      8      4      4      4      3      2      3      3      6   : 36688
aida06 :     14     17     10      3      3      3      1      4      3      3      6   : 37120
aida07 :      3      4      1      1      1      2      2      2      3      3      6   : 35740
aida08 :     12      5      7      2      2      4      1      2      3      3      6   : 35976
aida09 :      0      6      2      0      2      2      2      2      2      4      6   : 36816
aida10 :      4      2      5      2      1      4      2      4      2      3      6   : 36080
aida11 :     14      3      2      3      1      2      2      4      2      3      6   : 35856
aida12 :      1      2      0      2      1      4      1      3      3      3      6   : 36244

22:05 Stats OK (attach 5)
Temps OK (attach 6)
Bias OK (attach 7)

System wide checks as before - see above.

22:13 ADC calibration - Only 06,07 and 09 now fail
(TD re-calibrated remotely)

22:53 Beam off to allow access to S4 to reboot VME crate (akash).


22:58 Started compressing *old* runs on Screen 2, Workspace 5
nice -n10 gzip -v R[1-2][0-9]_* R[3][0-2]_*
aka everything *except* R33_*


23:11: Beam is back. Stats and UCESB as before.

23:24: Stats OK (attach 8)
Temps OK (attach 9)
Bias OK (attach 10)
UCESB OK (attach 11)
System-wide checks
Clock: 06 fails
Calibration: 06,07,09 fail

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd066 : 	 20  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

	 Base 		Current 	Difference
aida05 fault 	 0xdb9a : 	 0xdb9d : 	 3  
aida06 fault 	 0xb74f : 	 0xb752 : 	 3  
aida07 fault 	 0xd052 : 	 0xd066 : 	 20  
aida08 fault 	 0xe4c1 : 	 0xe4c4 : 	 3  
aida09 fault 	 0xaf4b : 	 0xaf4d : 	 2  
White Rabbit error counter test result: Passed 7, Failed 5

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15      8      2      2      3      4      3      3      2      3      6   : 35996
aida02 :      8     12      4      1      2      4      3      2      3      3      6   : 36448
aida03 :      2      4      2      4      3      3      1      3      3      3      6   : 36360
aida04 :      7      8      4      4      2      4      3      3      3      3      6   : 37020
aida05 :      3      5      7      4      3      3      3      3      3      3      6   : 36964
aida06 :      2      6      8      2      3      4      2      3      3      3      6   : 36792
aida07 :      8      4      5      2      0      3      1      2      3      4      6   : 37712
aida08 :     26     11      3      1      1      3      1      3      3      3      6   : 36304
aida09 :     22      4      2      2      3      4      1      3      3      3      6   : 36504
aida10 :     21      6      6      2      0      4      2      3      3      3      6   : 36644
aida11 :     22      7      2      3      2      1      2      4      2      3      6   : 35856
aida12 :     16      9      2      2      3      3      2      2      3      3      6   : 36136
Attachment 1: Stats1.png
Stats1.png
Attachment 2: Ucesb1.png
Ucesb1.png
Attachment 3: Temps1.png
Temps1.png
Attachment 4: Bias1.png
Bias1.png
Attachment 5: Stats2.png
Stats2.png
Attachment 6: Temp2.png
Temp2.png
Attachment 7: Bias2.png
Bias2.png
Attachment 8: Stats3.png
Stats3.png
Attachment 9: Temps3.png
Temps3.png
Attachment 10: Bias3.png
Bias3.png
Attachment 11: Ucesb2.png
Ucesb2.png
  331   Wed May 19 11:24:55 2021 RDPMonday 19 April 12 noon shift
System checks gave 11 passed/1 failed, except for ADC timimgs: 10 passed/2 failed.

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2


		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd056 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x16 : 	 22  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      2      0      5      5      1      2      3      3      6   : 36120
aida02 :     25     10      0      4      3      5      2      3      3      3      6   : 36980
aida03 :     22     12      5      2      1      3      2      3      3      3      6   : 36616
aida04 :     16      3      3      3      4      4      4      2      3      3      6   : 36840
aida05 :     14      5      6      2      3      4      4      2      3      3      6   : 36800
aida06 :     30     10      6      3      3      4      2      3      3      3      6   : 36936
aida07 :     27     11      1      0      0      3      2      3      3      3      6   : 36436
aida08 :     38      7      7      3      3      2      2      3      3      3      6   : 36704
aida09 :      4      6      4      2      4      4      1      4      3      3      6   : 37056
aida10 :     21     10      3      3      1      5      1      3      3      3      6   : 36596
aida11 :     24      8      2      1      4      3      2      4      2      3      6   : 36192
aida12 :     31     12      4      3      5      5      2      2      3      3      6   : 36668

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Mon Apr 19 10:32:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

There should be a separate elog note about changing one of the bias voltages.

All histograms zeroed at 16:13.

16.20 DAQ continues file NULL/R33_156
      aida09 HEC fast comparator 0x2->0x1
      DSSSD#3 bias -100->-120V
Attachment 1: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(3).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(3).png
Attachment 2: Screenshot_2021-04-19_Tape_Service_(Expert)_aidas-gsi.png
Screenshot_2021-04-19_Tape_Service_(Expert)_aidas-gsi.png
Attachment 3: 62.png
62.png
  244   Mon Apr 19 06:46:15 2021 MSMonday 19 April 07:47-12:00 2021

07:47

    
Clock status test result: Passed 12, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

    
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

    
         Base         Current     Difference
aida01 fault      0x7686 :      0x768a :      4  
aida02 fault      0x941d :      0x9421 :      4  
aida03 fault      0x7cd7 :      0x7cdb :      4  
aida04 fault      0xb86d :      0xb871 :      4  
aida05 fault      0x1a59 :      0x1a61 :      8  
aida06 fault      0x4f45 :      0x4f4d :      8  
aida07 fault      0x3bfc :      0x3c46 :      74  
aida08 fault      0xc7ce :      0xc7d5 :      7  
aida09 fault      0xb33b :      0xb33c :      1  
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

        
             Base         Current         Difference
aida09 fault      0x1 :      0x2 :      1  
aida12 fault      0xa :      0x35 :      43  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     19     18     10      7      1      3      2      4      2      3      6   : 36380
aida02 :     11     15      5      3      2      3      2      3      1      4      6   : 36692
aida03 :     15     11     13      3      4      3      3      3      2      3      6   : 36164
aida04 :     22     14     20      5      1      3      4      3      2      3      6   : 36456
aida05 :      7      7      5      2      1      2      3      4      2      3      6   : 36132
aida06 :     21     14      6      2      3      3      1      2      2      4      6   : 37028
aida07 :     11      8      7      4      2      3      2      2      4      3      6   : 37212
aida08 :     18     10      7      5      3      1      4      3      2      3      6   : 36072
aida09 :     24     23     18      3      2      3      2      4      2      3      6   : 36504
aida10 :      6     19      5      2      2      2      2      3      2      3      6   : 35520
aida11 :     22     15      7      2      1      3      2      3      2      3      6   : 35648
aida12 :     35     13      8      2      5      3      1      3      3      3      6   : 36724

 

At 8:30 DAQ and Temperature Scan stopped working properly.

 

 

11:00

It was noted that at the time that AIDA dropped out ~8:30 the beam also dropped out. It seems a bit of a coincidence

TD and MS powercycled the FEEs after the first crash and restarted MIDAS but not the merger. This recovered the FEEs but did not re-establish the links between the FEEs and the Merger.

Another powercycle was performed this time with a full reset of the merger and the links were restored. Upon restoring there was a large amount of noise in across all FEEs on average a 50% increase across all FEEs but in DSSD a factor of 4-6 increase was common.

In the waveforms large 100kHz transiets could be seen - attachment 4

A further power-cycle was performed by OH. The rates following this powercycle are better than just before the power cycle but have not recovered to pre-glitch levels.

The waveforms here are much improved though - attachment 5 and 6

 

 

11:45

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

 

 

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

 

    
White Rabbit error counter test result: Passed 12, Failed 0

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

 

    
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :      1      2      2      1      2      4      2      4      2      3      7   : 40148
aida02 :      5      4      2      2      1      2      2      3      2      3      7   : 39380
aida03 :      3      2      1      1      1      3      1      4      2      3      7   : 39692
aida04 :      4      2      1      0      1      3      3      3      2      3      7   : 39664
aida05 :      2      7      2      0      2      2      3      4      2      3      7   : 40160
aida06 :      2      3      2      2      2      2      1      3      3      3      7   : 40192
aida07 :     23      7      2      0      0      3      2      4      2      3      7   : 39988
aida08 :     23     11      4      0      1      3      1      4      2      3      7   : 39860
aida09 :      1      3      1      1      2      4      1      3      1      4      7   : 40396
aida10 :      1      3      3      1      1      4      1      2      3      3      7   : 39852
aida11 :      3      0      2      3      1      2      3      2      2      3      7   : 39116
aida12 :      5      8      0      3      1      4      2      3      2      3      7   : 39668

 

 

 

 

 

 

 

 

Attachment 1: Screenshot_2021-04-19_Statistics_aidas-gsi.png
Screenshot_2021-04-19_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: 42.png
42.png
Attachment 4: Screenshot_2021-04-19_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-04-19_Spectrum_Browser_aidas-gsi(1).png
Attachment 5: 210319_1054_Layout7.png
210319_1054_Layout7.png
Attachment 6: 210419_1054_Layout8.png
210419_1054_Layout8.png
Attachment 7: Screenshot_2021-04-19_Statistics_aidas-gsi(1).png
Screenshot_2021-04-19_Statistics_aidas-gsi(1).png
Attachment 8: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 9: 43.png
43.png
  320   Mon May 17 15:14:35 2021 LJWMonday 17th May 2021 16:00-20:00

16:25

Informed at shift handover that there has been no beam.

Statistics  - See attachment 1

Temperatures - See attachment 2

Bias & Leakage currents - See attachment 3

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

 

White Rabbit decoder status:

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4dd :      3  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

18:00

Usecb scalars now showing DSSD implant rates.

Statistics  - See attachment 4

Temperatures - See attachment 5

Bias & Leakage currents - See attachment 6

System Wide checks - All passed, EXCEPT:

ADC Calibration

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

White rabbit decoder status

   
         Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx

 

19:45

Usecb scalars have continued to show DSSD implant rates since the last recorded elog check.

Statistics  - See attachment 7

Temperatures - See attachment 8

Bias & Leakage currents - See attachment 9

System Wide checks - All passed, EXCEPT:

ADC Calibration:

   
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


White Rabbit decoder status:

     Base         Current     Difference
aida05 fault      0x4da :      0x4e1 :      7  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

 

19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes

19:55 - Beam now back again - DSSD implants showing

 

 

Attachment 1: 170521_1.png
170521_1.png
Attachment 2: 170521_2.png
170521_2.png
Attachment 3: 170521_3.png
170521_3.png
Attachment 4: 170521_4.png
170521_4.png
Attachment 5: 170521_5.png
170521_5.png
Attachment 6: 170521_6.png
170521_6.png
Attachment 7: 170521_7.png
170521_7.png
Attachment 8: 170521_8.png
170521_8.png
Attachment 9: 170521_9.png
170521_9.png
  321   Mon May 17 19:05:23 2021 LSMonday 17th May 20.00-24.00
20.30
Checked stats, seem okay (Fig 1)
Ran R14_99 through analyser, dead times all good (Fig 2)
Merger rate 1.8M items/s
Tape server 6.4MB/s

21.00
Checked stats seem okay (Fig 3)
Ran R14_106 through analyser, dead times all good (Fig 4)

21.30
Checked stats seem okay (Fig 5)
Ran R14_112 through analyser, dead times all good (Fig 6)

22.00
Stats okay (Fig 7)
FEE temps okay (Fig 8)
Rate spectra (Fig 9)
Leakage currents (Fig 10)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4e7 : 	 13  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.4M items/s
Tape server 6.3MB/s

Ran R14_119 through analyser, dead times all good (Fig 11)

22.30
Checked stats seem okay (Fig 12)
Ran R14_123 through analyser, dead times all good (Fig 13)

23.00
Checked stats seem okay (Fig 14)
Ran R14_129 through analyser, dead times all good (Fig 15)

23.30
Checked stats seem okay (Fig 16)
Ran R14_135 through analyser, dead times all good (Fig 17)

24.00
Stats okay (Fig 18)
FEE temps okay (Fig 19)
Leakage currents (Fig 20)

System wide checks:
Clock status all passed

White rabbit 1 fail:
		 Base 		Current 	Difference
aida05 fault 	 0x4da : 	 0x4eb : 	 17  
White Rabbit error counter test result: Passed 15, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA timestamp all passed

Merger rate 2.5M items/s
Tape server 6.5MB/s

Ran R14_142 through analyser, dead times all good (Fig 21)
Attachment 1: 210517_Stats_2026.png
210517_Stats_2026.png
Attachment 2: 210517_Deadtime_R14_99.png
210517_Deadtime_R14_99.png
Attachment 3: 210517_Stats_2101.png
210517_Stats_2101.png
Attachment 4: 210517_Deadtimes_R14_106.png
210517_Deadtimes_R14_106.png
Attachment 5: 210517_Stats_2128.png
210517_Stats_2128.png
Attachment 6: 210517_Deadtimes_R14_112.png
210517_Deadtimes_R14_112.png
Attachment 7: 210517_Stats_2159.png
210517_Stats_2159.png
Attachment 8: 210517_FEEtemps_2159.png
210517_FEEtemps_2159.png
Attachment 9: 210517_Spectra_2159.png
210517_Spectra_2159.png
Attachment 10: 210517_Bias_2159.png
210517_Bias_2159.png
Attachment 11: 210517_Deadtimes_R14_119.png
210517_Deadtimes_R14_119.png
Attachment 12: 210517_Stats_2230.png
210517_Stats_2230.png
Attachment 13: 210517_Deadtimes_R14_123.png
210517_Deadtimes_R14_123.png
Attachment 14: 210517_Stats_2300.png
210517_Stats_2300.png
Attachment 15: 210517_Deadtimes_R14_129.png
210517_Deadtimes_R14_129.png
Attachment 16: 210517_Stats_2328.png
210517_Stats_2328.png
Attachment 17: 210517_Deadtimes_R14_135.png
210517_Deadtimes_R14_135.png
Attachment 18: 210517_Stats_2359.png
210517_Stats_2359.png
Attachment 19: 210517_FEEtemps_2359.png
210517_FEEtemps_2359.png
Attachment 20: 210517_bias_2359.png
210517_bias_2359.png
Attachment 21: 210517_deadtime_R14_142_1.png
210517_deadtime_R14_142_1.png
  666   Mon Jun 17 02:13:21 2024 TDMonday 17 June
03.11 DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats OK - attachment 3

      per FEE64 Rate spectra - attachment 4

05.33 DSSSD bias & leakage current OK - attachment 5

      FEE64 temperatures OK - attachment 6

      ADC data item stats OK - attachment 7

      per FEE64 Rate spectra - attachment 8

      per FEE64 1.8.L spectra - attachments 9-11
       aida09 pulser walkthrough 60 ch FWHM

      per FEE64 1.8.W spectra - 20us FSR - attachments 12-14

      Merger, TapeServer etc - attachments 15-16

05.40 analysis data file R9_1042 - attachment 17

      Check ASIC load x2

      aida02 zero rate
      restart Merger

05.55 Data file S181/R10
      Pulser walkthrough

      All histograms zero'd

      BNC PB-5
      Amplitudes 10.0-1.0 @ 1.0 step
      Attenuation x10
      Polarity +
      tau_d 1ms
      Frequency 25Hz

      per p+n FEE64 1.8.L spectra - attachment 18

      per FEE64 Rate spectra - attachment 19

06.43 TapeServer no storage mode

09.14 Data file S181/R11
      Pulser walkthrough

      All histograms zero'd

      BNC PB-5
      Amplitudes 10.0-1.0 @ 1.0 step
      Attenuation x10
      Polarity -
      tau_d 1ms
      Frequency 25Hz

09.52 TapeServer no storage mode

      per n+n FEE64 1.8.L spectra - attachment 20

15.18 DEFGAS/FATIMA arrays moved downstream to permit installation of calibration source at centre of each array - increased distance to AIDA/bPlas => lower rate


15.16 DSSSD bias & leakage current OK - attachments 21-22

      FEE64 temperatures OK - attachment 23

      ADC data item stats OK - attachment 24
       aida02 zero rate

      per FEE64 Rate spectra - attachment 25

      Merger restart
      ADC data item stats OK - attachment 26
       7x < 20k, max 207k aida16

15.54 bPlast debiased.


16.29 Data file S181/R12
      bPlas OFF but remiains cabled/connected/grounded

16.30 DSSSD bias & leakage current OK - attachment 27

      FEE64 temperatures OK - attachment 28

      ADC data item stats OK - attachment 29

      per FEE64 Rate spectra - attachment 30

      per FEE64 1.8.W spectra - 20us, 200us, 2ms, 20ms FSR - see attachments 31-39
       note - not all FEE64s updated spectra for all settings

      ADC data item stats OK - attachment 40
       rates change cf. attachment 29
       sources installed/moved/removed, people moving around platform?

      per FEE64 Rate spectra - attachment 41

16.57 DSSSD bias OFF
      FEE64 power OFF

16.34 analysis data file S181/R12_7
      max deadtime 2.2% (aida08)
      ADC data rate 1.330M, HEC data rate 2.1k

16.52 analysis data file S181/R12_12
      max deadtime 1.7% (aida08)
      ADC data rate 945k, HEC data rate 1.8k
Attachment 1: Screenshot_from_2024-06-17_03-11-38.png
Screenshot_from_2024-06-17_03-11-38.png
Attachment 2: Screenshot_from_2024-06-17_03-12-05.png
Screenshot_from_2024-06-17_03-12-05.png
Attachment 3: Screenshot_from_2024-06-17_03-12-29.png
Screenshot_from_2024-06-17_03-12-29.png
Attachment 4: Screenshot_from_2024-06-17_03-12-53.png
Screenshot_from_2024-06-17_03-12-53.png
Attachment 5: Screenshot_from_2024-06-17_05-33-18.png
Screenshot_from_2024-06-17_05-33-18.png
Attachment 6: Screenshot_from_2024-06-17_05-33-47.png
Screenshot_from_2024-06-17_05-33-47.png
Attachment 7: Screenshot_from_2024-06-17_05-34-22.png
Screenshot_from_2024-06-17_05-34-22.png
Attachment 8: Screenshot_from_2024-06-17_05-34-53.png
Screenshot_from_2024-06-17_05-34-53.png
Attachment 9: Screenshot_from_2024-06-17_05-36-26.png
Screenshot_from_2024-06-17_05-36-26.png
Attachment 10: Screenshot_from_2024-06-17_05-38-40.png
Screenshot_from_2024-06-17_05-38-40.png
Attachment 11: Screenshot_from_2024-06-17_05-39-53.png
Screenshot_from_2024-06-17_05-39-53.png
Attachment 12: Screenshot_from_2024-06-17_05-40-29.png
Screenshot_from_2024-06-17_05-40-29.png
Attachment 13: Screenshot_from_2024-06-17_05-41-12.png
Screenshot_from_2024-06-17_05-41-12.png
Attachment 14: Screenshot_from_2024-06-17_05-42-07.png
Screenshot_from_2024-06-17_05-42-07.png
Attachment 15: Screenshot_from_2024-06-17_05-42-35.png
Screenshot_from_2024-06-17_05-42-35.png
Attachment 16: Screenshot_from_2024-06-17_05-43-06.png
Screenshot_from_2024-06-17_05-43-06.png
Attachment 17: R9_1042
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  259119494 ( 1239652.5 Hz)
        Other data format:    2800506 (   13397.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        509 (       2.4 Hz)
                               RESUME:        509 (       2.4 Hz)
                              SYNC100:      32779 (     156.8 Hz)
                              WR48-63:      32779 (     156.8 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    2733930 (   13079.4 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     425638 (    2036.3 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      209.026 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.013                0.000
  1                0.000                0.000
  2                0.006                0.000
  3                0.367                0.000
  4                0.000                0.000
  5                0.343                0.000
  6                0.000                0.000
  7                4.268                0.000
  8                0.000                0.000
  9                2.321                0.000
 10                0.126                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.030                0.000
 14                0.000                0.000
 15                1.478                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0    9607668       8795          0          0         21         21       1240       1240          0       6273          0          0
  1     568341        148          0          0          0          0         74         74          0          0          0          0
  2   11592594    1368472          0          0          2          2       1710       1710          0    1365048          0          0
  3   20057241       5226          0          0         32         32       2581       2581          0          0          0          0
  4    4038921       1042          0          0          0          0        521        521          0          0          0          0
  5   16834727       4288          0          0         27         27       2117       2117          0          0          0          0
  6    2895864      13265          0          0          0          0        361        361          0      12543          0          0
  7   58542191      15052          0          0        171        171       7355       7355          0          0          0     425638
  8    6363514       1604          0          0          0          0        802        802          0          0          0          0
  9   47066112    1362366          0          0        140        140       6010       6010          0    1350066          0          0
 10   19083458       4678          0          0         24         24       2315       2315          0          0          0          0
 11    2437006        602          0          0          0          0        301        301          0          0          0          0
 12    3839787        956          0          0          0          0        478        478          0          0          0          0
 13   12055396       3054          0          0         12         12       1515       1515          0          0          0          0
 14    3699941        872          0          0          0          0        436        436          0          0          0          0
 15   40436733      10086          0          0         80         80       4963       4963          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   69.695s (  459.141 blocks/s,  28.696 Mb/s)
Attachment 18: Screenshot_from_2024-06-17_06-44-16.png
Screenshot_from_2024-06-17_06-44-16.png
Attachment 19: Screenshot_from_2024-06-17_06-45-49.png
Screenshot_from_2024-06-17_06-45-49.png
Attachment 20: Screenshot_from_2024-06-17_09-53-53.png
Screenshot_from_2024-06-17_09-53-53.png
Attachment 21: Screenshot_from_2024-06-17_15-17-20.png
Screenshot_from_2024-06-17_15-17-20.png
Attachment 22: Screenshot_from_2024-06-17_15-16-47.png
Screenshot_from_2024-06-17_15-16-47.png
Attachment 23: Screenshot_from_2024-06-17_15-17-46.png
Screenshot_from_2024-06-17_15-17-46.png
Attachment 24: Screenshot_from_2024-06-17_15-18-09.png
Screenshot_from_2024-06-17_15-18-09.png
Attachment 25: Screenshot_from_2024-06-17_15-18-54.png
Screenshot_from_2024-06-17_15-18-54.png
Attachment 26: Screenshot_from_2024-06-17_15-20-50.png
Screenshot_from_2024-06-17_15-20-50.png
Attachment 27: Screenshot_from_2024-06-17_16-30-26.png
Screenshot_from_2024-06-17_16-30-26.png
Attachment 28: Screenshot_from_2024-06-17_16-30-55.png
Screenshot_from_2024-06-17_16-30-55.png
Attachment 29: Screenshot_from_2024-06-17_16-31-26.png
Screenshot_from_2024-06-17_16-31-26.png
Attachment 30: Screenshot_from_2024-06-17_16-31-48.png
Screenshot_from_2024-06-17_16-31-48.png
Attachment 31: Screenshot_from_2024-06-17_16-33-21.png
Screenshot_from_2024-06-17_16-33-21.png
Attachment 32: Screenshot_from_2024-06-17_16-34-32.png
Screenshot_from_2024-06-17_16-34-32.png
Attachment 33: Screenshot_from_2024-06-17_16-36-35.png
Screenshot_from_2024-06-17_16-36-35.png
Attachment 34: Screenshot_from_2024-06-17_16-37-33.png
Screenshot_from_2024-06-17_16-37-33.png
Attachment 35: Screenshot_from_2024-06-17_16-38-14.png
Screenshot_from_2024-06-17_16-38-14.png
Attachment 36: Screenshot_from_2024-06-17_16-40-21.png
Screenshot_from_2024-06-17_16-40-21.png
Attachment 37: Screenshot_from_2024-06-17_16-41-18.png
Screenshot_from_2024-06-17_16-41-18.png
Attachment 38: Screenshot_from_2024-06-17_16-43-19.png
Screenshot_from_2024-06-17_16-43-19.png
Attachment 39: Screenshot_from_2024-06-17_16-44-24.png
Screenshot_from_2024-06-17_16-44-24.png
Attachment 40: Screenshot_from_2024-06-17_16-45-19.png
Screenshot_from_2024-06-17_16-45-19.png
Attachment 41: Screenshot_from_2024-06-17_16-46-32.png
Screenshot_from_2024-06-17_16-46-32.png
Attachment 42: R12_7
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:      32000
          ADC data format:  258100864 ( 1330189.9 Hz)
        Other data format:    3819136 (   19682.9 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        522 (       2.7 Hz)
                               RESUME:        522 (       2.7 Hz)
                              SYNC100:      32723 (     168.6 Hz)
                              WR48-63:      32723 (     168.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    3752646 (   19340.2 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     409704 (    2111.5 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:      194.033 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.121                0.000
  1                0.000                0.000
  2                0.050                0.000
  3                0.191                0.000
  4                0.010                0.000
  5                0.209                0.000
  6                0.000                0.000
  7                4.225                0.000
  8                0.000                0.000
  9                2.072                0.000
 10                0.065                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.145                0.000
 14                0.002                0.000
 15                1.159                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0   13428500       9208          0          0         38         38       1659       1659          0       5814          0          0
  1    4595656       1164          0          0          0          0        582        582          0          0          0          0
  2   13843408    1881338          0          0         11         11       1937       1937          0    1877442          0          0
  3   19769270       5066          0          0         33         33       2500       2500          0          0          0          0
  4    5034345       1228          0          0          4          4        610        610          0          0          0          0
  5   19195180       4990          0          0         40         40       2455       2455          0          0          0          0
  6    5179175      12949          0          0          0          0        653        653          0      11643          0          0
  7   53830866      13806          0          0        173        173       6730       6730          0          0          0     409704
  8    6018644       1564          0          0          0          0        782        782          0          0          0          0
  9   41936601    1868943          0          0        126        126       5472       5472          0    1857747          0          0
 10   16378984       4032          0          0         16         16       2000       2000          0          0          0          0
 11    2633344        646          0          0          0          0        323        323          0          0          0          0
 12    1918841        428          0          0          0          0        214        214          0          0          0          0
 13   12464437       2916          0          0         14         14       1444       1444          0          0          0          0
 14    3882711       1026          0          0          1          1        512        512          0          0          0          0
 15   37990902       9832          0          0         66         66       4850       4850          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   60.555s (  528.448 blocks/s,  33.028 Mb/s)
Attachment 43: R12_12
 *** TDR format 3.3.0 analyser - TD - May 2021
 *** ERROR: READ I/O error:       5002
                   blocks:       7946
          ADC data format:   63712775 (  945168.0 Hz)
        Other data format:    1325235 (   19659.6 Hz)
 Sample trace data format:          0 (       0.0 Hz)
         Undefined format:          0 (       0.0 Hz)
   Other data format type:      PAUSE:        128 (       1.9 Hz)
                               RESUME:        128 (       1.9 Hz)
                              SYNC100:       8197 (     121.6 Hz)
                              WR48-63:       8197 (     121.6 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:    1308585 (   19412.6 Hz)
                           Other info:          0 (       0.0 Hz)

   ADC data range bit set:     119370 (    1770.8 Hz)

                Timewarps:        ADC:          0 (       0.0 Hz)
                                PAUSE:          0 (       0.0 Hz)
                               RESUME:          0 (       0.0 Hz)
                              SYNC100:          0 (       0.0 Hz)
                              WR48-63:          0 (       0.0 Hz)
                           FEE64 disc:          0 (       0.0 Hz)
                             MBS info:          0 (       0.0 Hz)
                            Undefined:          0 (       0.0 Hz)
                         Sample trace:          0 (       0.0 Hz)

 *** Timestamp elapsed time:       67.409 s
 FEE  elapsed dead time(s) elapsed idle time(s)
  0                0.000                0.000
  1                0.000                0.000
  2                0.000                0.000
  3                0.043                0.000
  4                0.000                0.000
  5                0.026                0.000
  6                0.000                0.000
  7                1.174                0.000
  8                0.000                0.000
  9                0.504                0.000
 10                0.000                0.000
 11                0.000                0.000
 12                0.000                0.000
 13                0.000                0.000
 14                0.000                0.000
 15                0.224                0.000
 16                0.000                0.000
 17                0.000                0.000
 18                0.000                0.000
 19                0.000                0.000
 20                0.000                0.000
 21                0.000                0.000
 22                0.000                0.000
 23                0.000                0.000
 24                0.000                0.000
 25                0.000                0.000
 26                0.000                0.000
 27                0.000                0.000
 28                0.000                0.000
 29                0.000                0.000
 30                0.000                0.000
 31                0.000                0.000
 32                0.000                0.000

 *** Statistics
 FEE  ADC Data Other Data     Sample  Undefined      Pause     Resume    SYNC100    WR48-63       Disc        MBS      Other   HEC Data
  0     896977       2260          0          0          0          0        119        119          0       2022          0          0
  1    1331618        334          0          0          0          0        167        167          0          0          0          0
  2    2033117     654263          0          0          0          0        346        346          0     653571          0          0
  3    6066546       1620          0          0         14         14        796        796          0          0          0          0
  4     632291        132          0          0          0          0         66         66          0          0          0          0
  5    5664311       1468          0          0         11         11        723        723          0          0          0          0
  6    1249421       4352          0          0          0          0        154        154          0       4044          0          0
  7   16999500       4406          0          0         59         59       2144       2144          0          0          0     119370
  8      43320         14          0          0          0          0          7          7          0          0          0          0
  9   12410393     652234          0          0         26         26       1617       1617          0     648948          0          0
 10    3213140        792          0          0          0          0        396        396          0          0          0          0
 11     278998         64          0          0          0          0         32         32          0          0          0          0
 12     433582        116          0          0          0          0         58         58          0          0          0          0
 13    1869315        430          0          0          0          0        215        215          0          0          0          0
 14     612732        128          0          0          0          0         64         64          0          0          0          0
 15    9977514       2622          0          0         18         18       1293       1293          0          0          0          0
 16          0          0          0          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0          0          0          0

 *** Timewarps
 FEE       ADC      Pause     Resume    SYNC100    WR48-63       Disc        MBS  Undefined    Samples
  0          0          0          0          0          0          0          0          0          0
  1          0          0          0          0          0          0          0          0          0
  2          0          0          0          0          0          0          0          0          0
  3          0          0          0          0          0          0          0          0          0
  4          0          0          0          0          0          0          0          0          0
  5          0          0          0          0          0          0          0          0          0
  6          0          0          0          0          0          0          0          0          0
  7          0          0          0          0          0          0          0          0          0
  8          0          0          0          0          0          0          0          0          0
  9          0          0          0          0          0          0          0          0          0
 10          0          0          0          0          0          0          0          0          0
 11          0          0          0          0          0          0          0          0          0
 12          0          0          0          0          0          0          0          0          0
 13          0          0          0          0          0          0          0          0          0
 14          0          0          0          0          0          0          0          0          0
 15          0          0          0          0          0          0          0          0          0
 16          0          0          0          0          0          0          0          0          0
 17          0          0          0          0          0          0          0          0          0
 18          0          0          0          0          0          0          0          0          0
 19          0          0          0          0          0          0          0          0          0
 20          0          0          0          0          0          0          0          0          0
 21          0          0          0          0          0          0          0          0          0
 22          0          0          0          0          0          0          0          0          0
 23          0          0          0          0          0          0          0          0          0
 24          0          0          0          0          0          0          0          0          0
 25          0          0          0          0          0          0          0          0          0
 26          0          0          0          0          0          0          0          0          0
 27          0          0          0          0          0          0          0          0          0
 28          0          0          0          0          0          0          0          0          0
 29          0          0          0          0          0          0          0          0          0
 30          0          0          0          0          0          0          0          0          0
 31          0          0          0          0          0          0          0          0          0
 32          0          0          0          0          0          0          0          0          0

 *** Program elapsed time:   14.254s (  557.461 blocks/s,  34.841 Mb/s)
  461   Mon May 16 15:12:50 2022 RDP, MAMonday 16th May 16:00-00:00

16:00 Took over the shift from Magda.

17:30 Updates: Attached (1-3)

19:30 Updates: Attached (4-6)

21:30 Updates: Attached (7-9)

23:00 Updates: Attached (10-12)

 

 

 

Attachment 1: Statistics_2022-05-16_17-32-40.png
Statistics_2022-05-16_17-32-40.png
Attachment 2: Temperature_2022-05-16_17-34-15.png
Temperature_2022-05-16_17-34-15.png
Attachment 3: LeakageCurrent_2022-05-16_17-35-05.png
LeakageCurrent_2022-05-16_17-35-05.png
Attachment 4: Statistics2022-05-16_19-31-28.png
Statistics2022-05-16_19-31-28.png
Attachment 5: Temperature2022-05-16_19-32-39.png
Temperature2022-05-16_19-32-39.png
Attachment 6: LeakageCurrent_2022-05-16_19-30-41.png
LeakageCurrent_2022-05-16_19-30-41.png
Attachment 7: Statistics2022-05-16_21-27-17.png
Statistics2022-05-16_21-27-17.png
Attachment 8: Temperature_2022-05-16_21-28-21.png
Temperature_2022-05-16_21-28-21.png
Attachment 9: LeakageCurrent2022-05-16_21-26-15.png
LeakageCurrent2022-05-16_21-26-15.png
Attachment 10: Statistics2022-05-16_22-59-14.png
Statistics2022-05-16_22-59-14.png
Attachment 11: Temperature_2022-05-16_23-00-18.png
Temperature_2022-05-16_23-00-18.png
Attachment 12: LeakageCurrent_2022-05-16_22-58-06.png
LeakageCurrent_2022-05-16_22-58-06.png
  459   Mon May 16 07:03:16 2022 MS, OHMonday 16th May 08:00-16:00

08:00 Took over the night shift from Tom (Attachments 1-3)

10:00: Attachement 4-6

around 10:30: Beam went off (Attachment 7)

11:00: Beam is back

12:00 Attachments 8-10

14:00 Attachements 11-13

15:50 Attachments 14-17

Attachment 1: Screenshot_2022-05-16_at_07-59-51_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_07-59-51_Statistics_aidas-gsi.png
Attachment 2: Screenshot_2022-05-16_at_08-00-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_08-00-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 3: Screenshot_from_2022-05-16_08-01-20.png
Screenshot_from_2022-05-16_08-01-20.png
Attachment 4: Screenshot_2022-05-16_at_09-59-15_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_09-59-15_Statistics_aidas-gsi.png
Attachment 5: Screenshot_2022-05-16_at_09-59-45_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_09-59-45_Temperature_and_status_scan_aidas-gsi.png
Attachment 6: Screenshot_from_2022-05-16_10-01-47.png
Screenshot_from_2022-05-16_10-01-47.png
Attachment 7: Screenshot_2022-05-16_at_10-47-01_ucesb.png
Screenshot_2022-05-16_at_10-47-01_ucesb.png
Attachment 8: Screenshot_2022-05-16_at_11-59-53_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_11-59-53_Statistics_aidas-gsi.png
Attachment 9: Screenshot_2022-05-16_at_12-00-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_12-00-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 10: Screenshot_from_2022-05-16_12-00-57.png
Screenshot_from_2022-05-16_12-00-57.png
Attachment 11: Screenshot_2022-05-16_at_14-00-56_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_14-00-56_Statistics_aidas-gsi.png
Attachment 12: Screenshot_2022-05-16_at_14-01-19_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_14-01-19_Temperature_and_status_scan_aidas-gsi.png
Attachment 13: Screenshot_from_2022-05-16_14-01-56.png
Screenshot_from_2022-05-16_14-01-56.png
Attachment 14: Screenshot_2022-05-16_at_15-47-58_Statistics_aidas-gsi.png
Screenshot_2022-05-16_at_15-47-58_Statistics_aidas-gsi.png
Attachment 15: Screenshot_2022-05-16_at_15-48-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2022-05-16_at_15-48-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 16: Screenshot_2022-05-16_at_15-48-34_Lost_activity_monitor_aidas-gsi.png
Screenshot_2022-05-16_at_15-48-34_Lost_activity_monitor_aidas-gsi.png
Attachment 17: Screenshot_from_2022-05-16_15-49-34.png
Screenshot_from_2022-05-16_15-49-34.png
  458   Sun May 15 23:00:08 2022 & TDMonday 16 May 00:00-08:00
00.01 DAQ continues
      file S450/R4_1319

      ADC control register 0xff

      all disc outputs disabled

      ASIC settings 2021Apr29-13-16-00
       slow comparator aida02 & aida04 0x16, aida06 & aida08 0x23, aida12 0xd, all others 0xc

      BNC PB-5 settings
      amplitude 1.000V
      attenuation x1
      tau_d 1ms
      frequency 2Hz
      polarity +

Disk space OK - /media/SecondDrive

[npg@aidas-gsi S450]$ df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 7.8G     0  7.8G   0% /dev
tmpfs                    7.8G  389M  7.4G   5% /dev/shm
tmpfs                    7.8G   19M  7.7G   1% /run
tmpfs                    7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root   50G   16G   35G  31% /
/dev/sda2               1014M  226M  789M  23% /boot
/dev/sda1                200M   12M  189M   6% /boot/efi
/dev/sde1                7.2T  4.1T  2.8T  61% /media/SecondDrive
/dev/mapper/centos-home  407G   91G  316G  23% /home
tmpfs                    1.6G   52K  1.6G   1% /run/user/1000
/dev/sdd1                7.2T  6.5T  310G  96% /run/media/npg/ThirdDrive

00.03 all histograms zero'd
      system wide checks counter baseline


00.08 check ASIC control - all FEE64s all ASICs

Attachments 1 & 2 - DSSSD bias & Leakage current - OK
                    grafana DSSSD bias, leakage current & temp - OK

Attachment 3 - FEE64 temps OK

Attachments 4-9 - adc, pause, resume & correlation scaler data items, push, flush stats

Attachments 10-16 - TapeServer, NewMerger, NewMerger stats

Attachment 17 - ucesb

04.08

Attachment 18 - DSSSD bias & Leakage current - OK

Attachment 19 - FEE64 temps OK

Attachment 20 - adc data item stats

Attachments 21 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6

06.36

Attachment 22 - DSSSD bias & Leakage current - OK

Attachment 23 - FEE64 temps OK

Attachment 24 - adc data item stats

Attachments 25 - ucesb

system wide checks - all OK *except* aida09 clock fail status 6
Attachment 1: Screenshot_from_2022-05-16_00-10-13.png
Screenshot_from_2022-05-16_00-10-13.png
Attachment 2: Screenshot_from_2022-05-16_00-10-43.png
Screenshot_from_2022-05-16_00-10-43.png
Attachment 3: Screenshot_from_2022-05-16_00-11-18.png
Screenshot_from_2022-05-16_00-11-18.png
Attachment 4: Screenshot_from_2022-05-16_00-11-50.png
Screenshot_from_2022-05-16_00-11-50.png
Attachment 5: Screenshot_from_2022-05-16_00-12-27.png
Screenshot_from_2022-05-16_00-12-27.png
Attachment 6: Screenshot_from_2022-05-16_00-12-54.png
Screenshot_from_2022-05-16_00-12-54.png
Attachment 7: Screenshot_from_2022-05-16_00-13-25.png
Screenshot_from_2022-05-16_00-13-25.png
Attachment 8: Screenshot_from_2022-05-16_00-13-55.png
Screenshot_from_2022-05-16_00-13-55.png
Attachment 9: Screenshot_from_2022-05-16_00-14-22.png
Screenshot_from_2022-05-16_00-14-22.png
Attachment 10: Screenshot_from_2022-05-16_00-15-46.png
Screenshot_from_2022-05-16_00-15-46.png
Attachment 11: Screenshot_from_2022-05-16_00-16-08.png
Screenshot_from_2022-05-16_00-16-08.png
Attachment 12: Screenshot_from_2022-05-16_00-16-36.png
Screenshot_from_2022-05-16_00-16-36.png
Attachment 13: Screenshot_from_2022-05-16_00-16-58.png
Screenshot_from_2022-05-16_00-16-58.png
Attachment 14: Screenshot_from_2022-05-16_00-16-58.png
Screenshot_from_2022-05-16_00-16-58.png
Attachment 15: Screenshot_from_2022-05-16_00-17-32.png
Screenshot_from_2022-05-16_00-17-32.png
Attachment 16: Screenshot_from_2022-05-16_00-17-55.png
Screenshot_from_2022-05-16_00-17-55.png
Attachment 17: Screenshot_from_2022-05-16_00-18-45.png
Screenshot_from_2022-05-16_00-18-45.png
Attachment 18: Screenshot_from_2022-05-16_04-08-34.png
Screenshot_from_2022-05-16_04-08-34.png
Attachment 19: Screenshot_from_2022-05-16_04-09-07.png
Screenshot_from_2022-05-16_04-09-07.png
Attachment 20: Screenshot_from_2022-05-16_04-09-57.png
Screenshot_from_2022-05-16_04-09-57.png
Attachment 21: Screenshot_from_2022-05-16_04-10-27.png
Screenshot_from_2022-05-16_04-10-27.png
Attachment 22: Screenshot_from_2022-05-16_06-35-32.png
Screenshot_from_2022-05-16_06-35-32.png
Attachment 23: Screenshot_from_2022-05-16_06-36-37.png
Screenshot_from_2022-05-16_06-36-37.png
Attachment 24: Screenshot_from_2022-05-16_06-37-04.png
Screenshot_from_2022-05-16_06-37-04.png
Attachment 25: Screenshot_from_2022-05-16_06-37-58.png
Screenshot_from_2022-05-16_06-37-58.png
  201   Mon Mar 15 11:47:51 2021 TDMonday 15 March
12.48 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

12.50 System wide checks

	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20     18      5      4      2      2      2      2      3      4     10   : 54576
aida02 :     26     18     16     10      6      4      2      4      2      3      6   : 37048
aida03 :     18     10      4      3      2      3      2      4      2      3      6   : 36152
aida04 :     13      9      4      2      2      2      2      4      1      4     15   : 73852
aida05 :     46     72     70     54     52     48     21     19      7      4      7   : 72216
aida06 :     32     10      4     10      0      2      2      4      3      2      5   : 31056
aida07 :     19      7      5      3      2      3      3      3      3      3      6   : 36916
aida08 :     20     14      2      3      1      3      1      4      3      3      6   : 36864
aida09 :      3      6      3      3      2      4      1      2      3      3      6   : 35916
aida10 :     23     11      3      3      1      3      3      2      3      3      6   : 36356
aida11 :      2      5      6      2      2      4      2      3      2      3      6   : 35664
aida12 :     25     10      5      2      2      2      2      3      3      3      6   : 36548


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 14Mb/s

12.55 Rate spectra - attachments 4 & 5

      Merger server error messages since most recent check

MERGE Data Link (20503): bad timestamp  3 3 0xc0d582ee 0x04f02352 0x000064aa97efd660 0x166c64aa94f02352 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e680cf 0x04f032f2 0x000064aa94f032f2 0x166c64aa94f032f2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c68215 0x04f03ac2 0x000064aa94f03ac2 0x166c64aa94f03ac2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd81f1 0x04f061d2 0x000064aa94f061d2 0x166c64aa94f061d2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0eb812a 0x04f07942 0x000064aa94f07942 0x166c64aa94f07942 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0f68041 0x04f07942 0x000064aa94f07942 0x166c64aa94f07942 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83806b78 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83811ea0 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c183d5 0x04f0aff2 0x000064aa94f0aff2 0x166c64aa94f0aff2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d58220 0x04f0b7c2 0x000064aa94f0b7c2 0x166c64aa94f0b7c2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e68097 0x04f0bf92 0x000064aa94f0bf92 0x166c64aa94f0bf92 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c682e7 0x04f0c762 0x000064aa94f0c762 0x166c64aa94f0c762 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cf7ff7 0x04f0cf32 0x000064aa94f0cf32 0x166c64aa94f0cf32 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd817e 0x04f0ee72 0x000064aa94f0ee72 0x166c64aa94f0ee72 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e180e5 0x04f105e2 0x000064aa94f105e2 0x166c64aa94f105e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d181fd 0x04f10db2 0x000064aa94f10db2 0x166c64aa94f10db2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d78072 0x04f11582 0x000064aa94f11582 0x166c64aa94f11582 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c17fff 0x04f14462 0x000064aa94f14462 0x166c64aa94f14462 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c68234 0x04f15bd2 0x000064aa94f15bd2 0x166c64aa94f15bd2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c780f9 0x04f17342 0x000064aa94f17342 0x166c64aa94f17342 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98277 0x04f17b12 0x000064aa94f17b12 0x166c64aa94f17b12 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d28174 0x04f17b12 0x000064aa94f17b12 0x166c64aa94f17b12 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d5812d 0x04f182e2 0x000064aa94f182e2 0x166c64aa94f182e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd8112 0x04f182e2 0x000064aa94f182e2 0x166c64aa94f182e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83806b78 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83811ea0 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d98224 0x04f18ab2 0x000064aa94f18ab2 0x166c64aa94f18ab2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e68096 0x04f19a52 0x000064aa94f19a52 0x166c64aa94f19a52 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c080e1 0x04f1a222 0x000064aa94f1a222 0x166c64aa94f1a222 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d18282 0x04f1a222 0x000064aa94f1a222 0x166c64aa94f1a222 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c38198 0x04f1a9f2 0x000064aa94f1a9f2 0x166c64aa94f1a9f2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cb8187 0x04f1b1c2 0x000064aa94f1b1c2 0x166c64aa94f1b1c2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc8178 0x04f1b992 0x000064aa94f1b992 0x166c64aa94f1b992 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c78063 0x04f1ffe2 0x000064aa94f1ffe2 0x166c64aa94f1ffe2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e980e1 0x04f1ffe2 0x000064aa94f1ffe2 0x166c64aa94f1ffe2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98129 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d28107 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0eb80e2 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660

18.47 FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 14Mb/s

      No new merger error messages since 12.55 today

          
Attachment 1: Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-03-15_Statistics_aidas-gsi.png
Screenshot_2021-03-15_Statistics_aidas-gsi.png
Attachment 3: 60.png
60.png
Attachment 4: Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi.png
Attachment 5: Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-03-15_Spectrum_Browser_aidas-gsi(1).png
Attachment 6: Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi(1).png
Screenshot_2021-03-15_Temperature_and_status_scan_aidas-gsi(1).png
Attachment 7: Screenshot_2021-03-15_Statistics_aidas-gsi(1).png
Screenshot_2021-03-15_Statistics_aidas-gsi(1).png
Attachment 8: 61.png
61.png
  368   Tue Jun 15 12:33:46 2021 TDMonday 14 June
  303   Mon May 10 08:49:37 2021 TDMonday 10 May
09.51 All system wide checks OK *except*


		 Base 		Current 	Difference
aida05 fault 	 0x8441 : 	 0x8442 : 	 1  
aida06 fault 	 0xc5c5 : 	 0xc5c6 : 	 1  
aida07 fault 	 0x6f07 : 	 0x6f08 : 	 1  
aida08 fault 	 0x4814 : 	 0x4815 : 	 1  
White Rabbit error counter test result: Passed 12, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      1      4      6      3      5      5      2      3      2      3      6   : 36004
aida02 :      4      1      3      6      3      4      3      3      2      3      6   : 36040
aida03 :     12      8      5      2      3      3      3      4      2      3      6   : 36416
aida04 :      1      2      5      4      1      4      3      4      2      3      6   : 36388
aida05 :     22      8      4      2      4      5      1      4      1      4      6   : 37272
aida06 :      5      4      0      6      3      6      3      3      2      3      6   : 36276
aida07 :     38     16      7      7      1      6      1      3      3      3      6   : 37032
aida08 :     16      4      2      3      1      4      1      3      3      3      6   : 36384
aida09 :      2      3      7      6      5      5      2      3      2      3      6   : 36112
aida10 :     42     15      4      6      2      6      2      4      2      3      6   : 36768
aida11 :      1      5     13      7      3      3      3      3      2      3      6   : 36124
aida12 :     23      9      3      4      7      3      2      3      2      3      6   : 35988
aida13 :      2      6      7      3      4      4      3      2      2      3      6   : 35592
aida14 :     17     11      3      4      3      5      2      3      2      3      6   : 35980
aida15 :      4      7      5      2      1      6      2      3      2      3      6   : 35864
aida16 :      4      1      3      3      4      2      3      4      2      3      6   : 36264


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Sun May 09 15:10:52 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:46 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:50 CEST 2021
 FEE : aida04 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:53 CEST 2021
 FEE : aida05 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:55 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:43:59 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:02 CEST 2021
 FEE : aida08 =>   Options file size is 1025 	 Last changed Wed May 05 12:15:54 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:08 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida12 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida13 =>   Options file size is 1025 	 Last changed Fri May 07 19:40:34 CEST 2021
 FEE : aida14 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida15 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021
 FEE : aida16 =>   Options file size is 1014 	 Last changed Thu Apr 29 14:44:57 CEST 2021

09.52 All histograms, stats and merger stats zero'd

      FEE64 temps OK - attachment 1
      ADC data, disc/WR 28-27/pause/correlation stats OK - attachments 2-6
      Rate spectra - attachment 7
       ASIC check load
      Rate spectra - attachment 8
      1.8.W spectra 20us FSR - attachments 9 & 10
      Grafana - most recent 7 days DSSSD bias and leakage current - attachment 11
      Merger stats/tape server/merger - attachment 12
      DSSSD bias and leakage current OK - attachment 13

10.08 1.8.W spectra 20us, 200us, 2ms & 20ms FSR - attachments 14-21
       we are not missing extrinsic noise features at longer time ranges cf. 20us FSR

12.18 Cannot connect to port 8015, MIDAS terminal says "Killed"
      Restart MIDAS 8015 
      Collect FPGA baseline to avoid TCL error
      FEEs unaffected and running as before

13.25 FEEs power cycled to see if any change in weird noise/rates situation
      No change 
      ASICs synchronised and back to normal

14.34 Start a pulser walkthrough in May21/R10
      Attenuation 5x, Rep Rate 25 Hz
      Amplitudes 10..9...1

14.45 Back to 2 Hz @ 2V Pulser, walkthrough complete. No Storage again.

 PULSER SETTINGS
---------------------------
 Pulse is ON
 Positive Tail Pulse
 Trigger Source is Internal Clock
 Trigger Threshold is 3.5
 Amplitude   : 2.0 Volts
 Rep Rate    : 2.0 hZ
 Delay       : 250.0 ns
 Fall Time   : 1 ms
 Attenuation : 1
 Display is  : Volts
 Equivalent keV is : 200.0 (NB. actually it is 12 MeV)
 Ramp Start at 0.01 Volts
 Ramp Stop  at 9.99 Volts
 Ramp Start at 1.0 keV
 Ramp Stop  at 999.0 keV
 Ramp Time  is 60 seconds
 # Ramp Cycles is 1

OK
Attachment 1: Screenshot_2021-05-10_Temperature_and_status_scan_aidas-gsi.png
Screenshot_2021-05-10_Temperature_and_status_scan_aidas-gsi.png
Attachment 2: Screenshot_2021-05-10_Statistics_aidas-gsi.png
Screenshot_2021-05-10_Statistics_aidas-gsi.png
Attachment 3: Screenshot_2021-05-10_Statistics_aidas-gsi(1).png
Screenshot_2021-05-10_Statistics_aidas-gsi(1).png
Attachment 4: Screenshot_2021-05-10_Statistics_aidas-gsi(2).png
Screenshot_2021-05-10_Statistics_aidas-gsi(2).png
Attachment 5: Screenshot_2021-05-10_Statistics_aidas-gsi(3).png
Screenshot_2021-05-10_Statistics_aidas-gsi(3).png
Attachment 6: Screenshot_2021-05-10_Statistics_aidas-gsi(4).png
Screenshot_2021-05-10_Statistics_aidas-gsi(4).png
Attachment 7: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi.png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi.png
Attachment 8: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(1).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(1).png
Attachment 9: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(2).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(2).png
Attachment 10: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(3).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(3).png
Attachment 11: Screenshot_2021-05-10_AIDA_Alerting_-_Grafana.png
Screenshot_2021-05-10_AIDA_Alerting_-_Grafana.png
Attachment 12: 510.png
510.png
Attachment 13: 511.png
511.png
Attachment 14: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(4).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(4).png
Attachment 15: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(11).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(11).png
Attachment 16: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(5).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(5).png
Attachment 17: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(10).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(10).png
Attachment 18: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(6).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(6).png
Attachment 19: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(9).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(9).png
Attachment 20: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(7).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(7).png
Attachment 21: Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(8).png
Screenshot_2021-05-10_Spectrum_Browser_aidas-gsi(8).png
  701   Tue Mar 11 11:53:18 2025 TDMonday 10 March
18.30 AIDA powered up for brief test

      DSSSD bias & leakage current OK - attachment 1

      FEE64 temperatures OK - attachment 2

      ADC data item stats - attachment 3
       ASIC settings at power up - slow comparator 0x64

      per p+n FEE64 1.8.L spectra - attachment 4
       pulser peak width aida9 55 ch FWHM (39keV FWHm), aida15 53 ch FWHM (37keV FWHM)

      p+n FEE64 aida15 1.8.W spectra - 20us FSR - attachments 5-6

      saved *.L, *.H, *.W spectra for aida15 - attachments 7-11
       *.W spectra - 20us, 200us, 2ms & 20ms FSR 


19.27 DAQ STOP
      DSSSD bias OFF
      FEE64 power OFF
Attachment 1: Screenshot_from_2025-03-10_18-41-32.png
Screenshot_from_2025-03-10_18-41-32.png
Attachment 2: Screenshot_from_2025-03-10_18-42-02.png
Screenshot_from_2025-03-10_18-42-02.png
Attachment 3: Screenshot_from_2025-03-10_18-42-33.png
Screenshot_from_2025-03-10_18-42-33.png
Attachment 4: Screenshot_from_2025-03-10_18-43-07.png
Screenshot_from_2025-03-10_18-43-07.png
Attachment 5: Screenshot_from_2025-03-10_18-48-03.png
Screenshot_from_2025-03-10_18-48-03.png
Attachment 6: Screenshot_from_2025-03-10_18-48-39.png
Screenshot_from_2025-03-10_18-48-39.png
Attachment 7: 2025Mar10-17.59.58.tar.gz
Attachment 8: 2025Mar10-18.05.01.tar.gz
Attachment 9: 2025Mar10-18.06.29.tar.gz
Attachment 10: 2025Mar10-18.12.03.tar.gz
Attachment 11: 2025Mar10-18.17.55.tar.gz
  480   Mon Jun 13 16:38:37 2022 NHMon 13 Jun
Remove Caen HV Supply

Open up, C23 was grounded (HV grounded in NIM)
Remove C23 (HV floating at NIM) and reinstall

Rebias HV... No Change

Reboot AIDA... No change
  652   Tue Jun 11 18:45:37 2024 PPMid-shift checks, 19:45

All looks good.

Sceenshots attached.

Attachment 1: Screenshot_from_2024-06-11_19-46-41.png
Screenshot_from_2024-06-11_19-46-41.png
Attachment 2: Screenshot_from_2024-06-11_19-47-09.png
Screenshot_from_2024-06-11_19-47-09.png
Attachment 3: Screenshot_from_2024-06-11_19-47-38.png
Screenshot_from_2024-06-11_19-47-38.png
Attachment 4: Screenshot_from_2024-06-11_19-48-12.png
Screenshot_from_2024-06-11_19-48-12.png
Attachment 5: Screenshot_from_2024-06-11_19-48-58.png
Screenshot_from_2024-06-11_19-48-58.png
Attachment 6: Screenshot_from_2024-06-11_19-49-22.png
Screenshot_from_2024-06-11_19-49-22.png
Attachment 7: Screenshot_from_2024-06-11_19-49-43.png
Screenshot_from_2024-06-11_19-49-43.png
Attachment 8: Screenshot_from_2024-06-11_19-50-19.png
Screenshot_from_2024-06-11_19-50-19.png
ELOG V3.1.3-7933898