21.19 system wide checks

FEE64 module aida06 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida06 fault 	 0x679f : 	 0x67a2 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      7      5      7      3      3      2      4      2      3      6   : 36356
aida02 :     67     29     27      4      0      4      2      3      2      3      4   : 28196
aida03 :     14      5      2      4      3      5      2      2      3      3      6   : 36416
aida04 :      5      5      0      7      2      3      3      3      3      3      6   : 36892
aida05 :     26      5      4      3      3      3      1      3      3      3      6   : 36464
aida06 :     20      7      5      2      3      4      2      2      3      3      6   : 36312
aida07 :     24      7      1      2      3      4      2      3      3      3      6   : 36776
aida08 :     22      7      6      2      3      3      1      3      3      3      6   : 36464
aida09 :     20      7      1      2      1      3      3      2      3      3      6   : 36248
aida10 :      9      1      6      1      1      2      2      4      2      3      6   : 35820
aida11 :     15      4      2      3      3      4      3      3      2      3      6   : 35996
aida12 :     22     11      6      2      2      3      2      2      3      3      6   : 36176


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1026 	 Last changed Fri Apr 23 03:51:08 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

21.20 DSSSD biases & leakage currents OK - attachment 1
      FEE64 tempos OK - attachment 2
      good event stats - attachment 3
      Grafana DSSSD biases & leakage currents for most recent 7 days - attachment 4

23.35 System wide checks - no changes to above
      DSSSD biases & leakage currents OK - attachment 5
      FEE64 tempos OK - attachment 6
      good event stats - attachment 7

23.46 DAQ continues file S460/R51_378