02:14 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at 3M items/sec
      tape service running at 26MB/sec

All attached to this entry with screenshots to 1-7

04:05 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at 3M items/sec
      tape service running at 26MB/sec

All attached tp this entry with screenshots 7-14

02:14 all system wide checks okay 
      FEE temperatures okay 
      leakage currents okay and recorded to spreadsheet 
      good event stats okay 
      merger running at around 3M items/sec
      tape service running at 26MB/sec

All attached to this entry with screenshots to 1-7

Elog went down around the end of the last shift, so final screenshots were not uploaded at that point.

However, AIDA appears to have contuned running smoothly in the interim (at least until 22:00).

Attachments 1 - 6 were taken around 16:00, 7 - 13 at around 18:00 , 14 - 18 at around 21:00.

Another set of sceenshots taken at around 23:00 - attachments 19 - 26.

16:00 Took over the shift from Magda.

17:30 Updates: Attached (1-3)

19:30 Updates: Attached (4-6)

21:30 Updates: Attached (7-9)

23:00 Updates: Attached (10-12)

12.30 There appear to be fewer HEC events in aida09 cf. aida11 and upstream DSSSDs
      Decrease aida09 asic #1-4 HEC fast comparator 0x2 -> 0x1
      Online/nearline analyses of daat files indicates that this does not resolve the issue

Some timestamp issues 

MERGE Data Link (23725): bad timestamp  11 3 0x8b620008 0x0287be94 0x00003dd78287be94 0x16773dd78287be94 0x16773dd78287beb2
MERGE Data Link (23725): bad timestamp  11 3 0x8b610040 0x073302ce 0x00003df3e73302ce 0x16773df3e73302ce 0x16773df3e73302ec
MERGE Data Link (23725): bad timestamp  11 3 0x8b610020 0x0a54b794 0x00003e4e4a54b794 0x16773e4e4a54b794 0x16773e4e4a54b7b2
MERGE Data Link (23725): bad timestamp  11 3 0x8b610020 0x052da21e 0x00003f68f52da21e 0x16773f68f52da21e 0x16773f68f52da232
MERGE Data Link (23725): bad timestamp  11 3 0x8b620008 0x0061489e 0x000040aa8061489e 0x167740aa8061489e 0x167740aa806148a8
MERGE Data Link (23725): bad timestamp  11 3 0x8b610008 0x04029d5e 0x00004169c4029d5e 0x16774169c4029d5e 0x16774169c4029d68
MERGE Data Link (23725): bad timestamp  11 3 0x8b630004 0x08dfcb14 0x0000417fd8dfcb14 0x1677417fd8dfcb14 0x1677417fd8dfcb28
MERGE Data Link (23725): bad timestamp  11 3 0x8b630020 0x08dfcb1e 0x0000417fd8dfcb1e 0x1677417fd8dfcb1e 0x1677417fd8dfcb28

13.50 Increased DSSSD# 3 bias from -100V

      -100V -6.325uA
      -110V -6.500uA
      -120V -6.800uA



       
      

System checks gave 11 passed/1 failed, except for ADC timimgs: 10 passed/2 failed.

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2


		 Base 		Current 	Difference
aida07 fault 	 0xd052 : 	 0xd056 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x16 : 	 22  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      2      6      2      0      5      5      1      2      3      3      6   : 36120
aida02 :     25     10      0      4      3      5      2      3      3      3      6   : 36980
aida03 :     22     12      5      2      1      3      2      3      3      3      6   : 36616
aida04 :     16      3      3      3      4      4      4      2      3      3      6   : 36840
aida05 :     14      5      6      2      3      4      4      2      3      3      6   : 36800
aida06 :     30     10      6      3      3      4      2      3      3      3      6   : 36936
aida07 :     27     11      1      0      0      3      2      3      3      3      6   : 36436
aida08 :     38      7      7      3      3      2      2      3      3      3      6   : 36704
aida09 :      4      6      4      2      4      4      1      4      3      3      6   : 37056
aida10 :     21     10      3      3      1      5      1      3      3      3      6   : 36596
aida11 :     24      8      2      1      4      3      2      4      2      3      6   : 36192
aida12 :     31     12      4      3      5      5      2      2      3      3      6   : 36668

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Mon Apr 19 10:32:22 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

There should be a separate elog note about changing one of the bias voltages.

All histograms zeroed at 16:13.

16.20 DAQ continues file NULL/R33_156
      aida09 HEC fast comparator 0x2->0x1
      DSSSD#3 bias -100->-120V

This appeared in the system log:

Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors

These messages seem to appear sporadically.

18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?

18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.

aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters

Thanks Nic!

22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)

22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.

At 20:15, We lost connection with AnyDesk.

Liliana restarted Anydesk on the AIDA DAQ PC. All seemed fine on AIDA DAQ but there was a more generic issue on the main DAQ, possibly the timesorter.

Helena restarted all after ~30mn and it's all working again now.

Wide checks at Time 22:22 CET

Clock status test result: Passed 16, Failed 0

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

         Base         Current     Difference
aida01 fault      0xf932 :      0xf933 :      1  
aida02 fault      0x62ec :      0x62ed :      1  
aida03 fault      0x8679 :      0x867a :      1  
aida04 fault      0xf0e4 :      0xf0e5 :      1  
aida05 fault      0x9db8 :      0x9dbc :      4  
aida06 fault      0x7f18 :      0x7f19 :      1  
aida07 fault      0xdd2c :      0xdd2d :      1  
aida08 fault      0x1557 :      0x1558 :      1  
White Rabbit error counter test result: Passed 8, Failed 8

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

             Base         Current         Difference
aida05 fault      0x0 :      0x2 :      2  
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :    4    8    16    32    64    128    256    512    1k    2k    4k
aida01 :     35     11      3      5      5      4      1      3      2      4      6   : 37876
aida02 :     20      5      4      3      2      5      2      4      3      3      6   : 37400
aida03 :     20      3      4      3      4      4      1      3      2      4      6   : 37640
aida04 :     25      6      2      3      4      4      3      3      3      3      6   : 37140
aida05 :     30     10      5      7      2      6      2      4      2      4      6   : 38776
aida06 :     21     11      4      4      3      5      2      4      3      3      6   : 37548
aida07 :     23      8      4      4      2      6      1      3      2      4      6   : 37852
aida08 :     27      7      2      3      4      3      2      4      3      3      6   : 37284
aida09 :     31      6      5      6      3      3      2      4      3      3      6   : 37372
aida10 :     22      5      3      4      3      6      2      2      2      4      6   : 37616
aida11 :     23      5      4      2      5      2      1      3      2      4      6   : 37444
aida12 :     20      7      4      7      3      5      2      3      3      3      6   : 37096
aida13 :     28      5      3      3      1      5      2      3      3      3      6   : 36840
aida14 :     38     12      2      4      2      4      1      4      3      3      6   : 37144
aida15 :     27      9      5      4      2      5      3      2      3      3      6   : 36740
aida16 :     21      6      4      4      1      4      2      3      3      3      6   : 36740

16:20
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc525 : 	 39  
aida08 fault 	 0xf0e9 : 	 0xf166 : 	 125  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
                 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x23 : 	 9  

Temps OK (attachment 1), Stats OK (attachment 2), Bias and leakage OK (attachment 3)


18:05
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc527 : 	 41  
aida08 fault 	 0xf0e9 : 	 0xf177 : 	 142  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x1c : 	 11  
aida08 fault 	 0x1a : 	 0x24 : 	 10  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 4), Stats OK (attachment 5), Bias and leakage OK (attachment 6)


20:22
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf17f : 	 150  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x20 : 	 15  
aida08 fault 	 0x1a : 	 0x25 : 	 11  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 7), Stats OK (attachment 8), Bias and leakage OK (attachment 9)

21:56
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc529 : 	 43  
aida08 fault 	 0xf0e9 : 	 0xf186 : 	 157  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
		 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x26 : 	 12  

Temps OK (attachment 10), Stats OK (attachment 11), Bias and leakage OK (attachment 12)

23:50
System wide checks.
Clock status OK
White Rabbit Error status:
		 Base 		Current 	Difference
aida07 fault 	 0xc4fe : 	 0xc52e : 	 48  
aida08 fault 	 0xf0e9 : 	 0xf18c : 	 163  
White Rabbit error counter test result: Passed 6, Failed 2

FPGA error status:	
	
			 Base 		Current 		Difference
aida07 fault 	 0x11 : 	 0x21 : 	 16  
aida08 fault 	 0x1a : 	 0x27 : 	 13  
FPGA Timestamp error counter test result: Passed 6, Failed 2

Temps OK (attachment 13), Stats OK (attachment 14), Bias and leakage OK (attachment 15)

> All 12 FEEs have valid WR Timestamps
> Had to powercycle aida09 once as before raw readout was displaying upper 12 bits of WR timestamp as 0. Unsure of other method.
> 
> HDMI cables in aida09 checked and good.

The problem would be the cable , one end or the other. 
I think ( if I recall ) a setup would restart the WR decoder.

I notice you have set the WR info word rate to be quite high , 6123/sec typ, is this intentional ?

All seems OK

Screenshots attached.

17.33 per FEE64 1.8.W spectra - attachments 8-9
      per p+n FEE64 1.8.L spectra - attachment 10
       aida09 pulser peak width 66 ch FWHM
       cf 
       c. 11:00 today 57 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/650
       cf
       c. 00:10 today 53 ch FWHM            https://elog.ph.ed.ac.uk/DESPEC/649

17.48 ADC data item stats - attachment 11
      All histograms zero'd

17.51 Analysis data file S181/R6_96 - attachment 12
      max deadtime 5% (aida08)
      HEC Data items 3.7kHz, LEC data items 1.8MHz
      cf.
      c. 12.29 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/650
      c. 00.09 today 1.1MHz    https://elog.ph.ed.ac.uk/DESPEC/649

4:13

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

Screenshot 1: statistics

Screenshot 2: temperatures

Screenshot 3: scalers

Screenshot 4: Bias and leakage current

6:36

AIDA stats OK

Leakage current OK

Temperatures OK

grafana OK

ucesb rates OK

System wide check done and same results as earlier: aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR

Screenshot 5: statistics

Screenshot 6: temperatures

Screenshot 7: scalers

Screenshot 8: Bias and leakage current

Checks 08:00-08:30

Screenshots are attached and everything looks OK.

Aida02 is down. Contacted Nick.

The rest looks stable, see screenshots.

10:53 CEST (NH): Rebooted aida02 via telnet
Resync ASIC clocks
All system wide checks look ok
ASIC threshold => 0x14

Go > All 16 writing to merger, looks OK

aida02 went down again 11:35 CEST
Rebooted with same procedure and is back again

Nick rebooted AIDA and aida02 came back at around 10:55.

AIDA was out of the DAQ from about 10:30 until 10:55

We lost aida02 at about 11:25 again. Nick fixed it about 11:50.

It is now back in the DAQ.

Everything is OK now.

Screenshots attached.

All looks good.

Screenshots attached

All looks good.

Screenshots attached.