08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151

Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
It is also the least noisy FEE in the back DSSD

MERGE Data Link (20510): bad timestamp  10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
MERGE Data Link (20510): bad timestamp  10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262

09:01 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Statistics - attachment 1
     Temperature - attachment 2
     Bias and leakage current - attachment 3

11:18 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c78 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics - attachment 4
     Temperatures - attachment 5
     Bias and leakage - attachment 6


13.14 DAQ continues OK - file R56_285

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

13.18 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c78 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      8      3      4      2      2      2      2      3      4     10   : 54492
aida02 :      0      2      3      0      2      3      2      3      3      3      6   : 36416
aida03 :     18      9      1      2      0      4      1      3      3      3      6   : 36320
aida04 :      2      3      3      3      1      4      3      2      2      4     15   : 74224
aida05 :     26     72     72     54     53     49     22     18      7      4      7   : 72104
aida06 :      7      5      5     10      0      2      2      4      3      2      5   : 30932
aida07 :      2      4      2      1      1      3      2      3      2      3      7   : 39464
aida08 :     25      9      4      2      1      2      1      4      3      3      6   : 36716
aida09 :      4     11      3      5      2      4      1      2      3      3      6   : 36024
aida10 :     18      5      2      1      0      5      2      2      3      3      6   : 36144
aida11 :     25     10      4      1      1      4      2      3      2      3      6   : 35668
aida12 :     26      7      1      1      1      3      2      3      3      3      6   : 36496

13.19 FEE64 temperatures OK - attachment 7
      Good events stats OK - attachment 8
      Bias & leakage currents OK - attachment 9
      Merger 4.5M items/s
      Tape Server 14Mb/s
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      All histograms zero'd


16.00 DAQ contiues R56_361

       System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c79 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      8      6      4      2      2      2      2      3      4     10   : 54524
aida02 :      1      3      4      2      1      2      1      4      3      3      6   : 36572
aida03 :      2      3      2      3      1      3      2      2      3      3      6   : 35936
aida04 :     19      7      2      3      1      4      3      3      1      4     15   : 73796
aida05 :     34     68     72     54     53     49     22     18      7      4      7   : 72104
aida06 :     11     10      4     10      0      2      2      4      3      2      5   : 30972
aida07 :      1      3      2      2      2      2      2      3      2      3      7   : 39420
aida08 :     24     10      5      2      1      3      1      4      3      3      6   : 36864
aida09 :     20      8      3      5      2      3      2      3      3      3      6   : 36704
aida10 :      2      3      2      3      1      3      3      2      3      3      6   : 36192
aida11 :     17     10      4      0      1      4      2      3      2      3      6   : 35604
aida12 :     20      6      3      2      1      3      2      2      2      4      6   : 37040


15.05 FEE64 temperatures OK - attachment 10
      Good events stats OK - attachment 11
      Bias & leakage currents OK - attachment 12
      Merger 4.6M items/s
      Tape Server 14Mb/s
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      Rate spectra - attachments 13 & 14
      p+n junction HEC spectra - attachment 15

17.33 end of S452 run file R56_397
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      BNC PB-5 config - attachment 16
      CAEN 8xxx NIM bin config - attachment 17