00:08 DAQ continues ok - writing to file R46_221

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a4 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      6      2      5      1      3      2      2      3      4     10   : 54492
aida02 :     12      7      3      4      3      3      3      2      3      3      6   : 36440
aida03 :      4      4      2      4      2      3      2      2      3      3      6   : 36048
aida04 :     18      9      4      2      1      3      2      4      1      4     15   : 73936
aida05 :     31     11      8      7      6      4      2     18      7      4      7   : 55220
aida06 :     36      8     18      4      4      2      1      4      3      2      5   : 31088
aida07 :     26      7      4      3      1      2      1      3      3      3      6   : 36224
aida08 :     25     12      5      4      4      3      2      3      3      3      6   : 36884
aida09 :      2      9      4      2      2      2      2      3      3      3      6   : 36432
aida10 :     11      3      4      6      4      3      1      3      3      3      6   : 36548
aida11 :     21      8      5      0      1      3      3      3      2      3      6   : 35748
aida12 :     10      6      6      2      4      4      1      3      3      3      6   : 36600

00:14 FEE64 Temperatures ok - attachment 1
      Good event statistics ok - attachment 2
      Detector bias/leakage currents ok - attachment 3

00:17 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

01:30 beam off

01:31 beam back

02:19 system wide checks

      same as entry at 00:09, *except*

 		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a5 : 	 5  
White Rabbit error counter test result: Passed 11, Failed 1

02:22 FEE64 Temperatures ok - attachment 4
      Good event statistics ok - attachment 5
      Detector bias/leakage currents ok - attachment 6

02:30 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

      no further bad timestamp errors in NewMerger terminal

02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7

02:50 ... and sure enough lose connection to aida07 after checking stats

      DAQ stop, Merger/TapeServer closed

      telnet aida07 -> sent reboot command

      mounted ok, reset DAQ, TapeService and Merge

      restarted DATA forwarding to MBS

      all looks to be ok now -> writing to R56

03:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	        Base 	Current Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5      3      7      2      2      2      2      3      4     10   : 54508
aida02 :      2      7      3      2      2      3      1      4      3      3      6   : 36784
aida03 :      8      8      1      3      2      3      1      3      3      3      6   : 36304
aida04 :     20      9      2      3      0      3      2      4      1      4     15   : 73880
aida05 :     51     74     70     54     53     49     22     18      7      4      7   : 72188
aida06 :     15     15      3      9      1      2      2      4      3      2      5   : 31044
aida07 :     24      6      2      3      3      4      2      3      2      3      7   : 39888
aida08 :     19     15      1      2      1      3      1      4      3      3      6   : 36820
aida09 :     18      4      2      7      3      4      1      3      3      3      6   : 36648
aida10 :     20      7      1      1      1      3      3      2      3      3      6   : 36216
aida11 :     25      7      5      3      4      2      1      4      2      3      6   : 35916
aida12 :      3      3      2      2      3      2      3      2      3      3      6   : 36164

03:18 FEE64 Temperatures ok - attachment 8
      Good event statistics ok - attachment 9
      Detector bias/leakage currents ok - attachment 10

      Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s
      
03:29 more bad timestamp error messages - attachment 11

03:52 analyser output on R56_35 - attachment 12

      no timewarps

03:53 DAQ continues ok, writing to file R56_41

05:18 system wide checks:

      same as entry at 03:08, *but* with additional WR decoder status error

Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

05:19 FEE64 Temperatures ok - attachment 14
      Good event statistics ok - attachment 13
      Detector bias/leakage currents ok - attachment 15

      Merger ok - 4.4M items/s
      TapeServer ok - 15 MB/s

05:25 rate spectra - attachment 16

05:33 AIDA implant XY hit patterns - attachments 17/18/19

05:56 BuTIS interface control set to 0x3 at 04.52 UTC

07:06 system wide checks - same error messages as entry at 05:18

07:08 FEE64 Temperatures ok - attachment 20

      good event statistics ok - attachment 21

      detector bias/leakage currents ok - attachment 22

      Merger ok - 4.3M items/s
      TapeServer ok - 15 MB/s

07:22 no bad timestamp errors since 02:30 UTC