18.00 DAQ continues OK - file R46_64

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

18.02 System wide checks OK *except*

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 16Mb/s

      All histograms zero'd

17.15 Most recent messages in merger server terminal session

MERGE Data Link (2322): bad timestamp  6 3 0xc1b57e35 0x0ba1685c 0x0000ef3fbba1685c 0x166bef3fbba1685c 0x166bef3fbba1879c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a77dc2 0x0ba1702c 0x0000ef3fbba1702c 0x166bef3fbba1702c 0x166bef3fbba1879c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bf7f66 0x0ba1702c 0x0000ef3fbba1702c 0x166bef3fbba1702c 0x166bef3fbba1879c
Transfer Error - : Broken pipe
Error 32
1: send() failed:
TCP transfer library version 4.0
1: TCP socket send buffer was 16384 - now 249856
1: TCP socket receive buffer was 87380 - now 249856
1: TCP socket created OK - now connecting to localhost port 10305
1: Connected to localhost port 10305


22.05 DAQ continues OK - file R46_168

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a2 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     18      6      7      5      1      3      1      3      3      4     10   : 54856
aida02 :     19      6      4      3      4      4      2      3      3      3      6   : 36892
aida03 :     22      8      1      2      1      2      1      3      3      3      6   : 36136
aida04 :     24      8      4      2      1      3      2      4      1      4     15   : 73952
aida05 :     37     14      9      8      7      3      2     18      7      4      7   : 55252
aida06 :     28     12     16      4      4      2      1      4      3      2      5   : 31056
aida07 :     28      9      2      2      2      2      1      3      3      3      6   : 36248
aida08 :     21     12      6      4      3      4      2      3      3      3      6   : 36948
aida09 :     18      5      2      1      1      3      2      2      3      3      6   : 35952
aida10 :     17      8      4      5      3      3      1      3      3      3      6   : 36516
aida11 :     21      8      5      0      0      4      3      3      2      3      6   : 35812
aida12 :     13     11      3      3    


      FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.4M data items/s
      TapeServer OK - 15Mb/s

22.17 Rate spectra - attachments 7 & 8
      p+n junction HEC spectra - attachment 9

      Merger server error messages since 17.15

MERGE Data Link (2316): bad timestamp  0 3 0x80300000 0x0164e7f8 0x000000000164e7f8 0x166200000164e7f8 0x166bffffffa7654e
MERGE Data Link (2324): bad timestamp  8 3 0x88300000 0x0534e25c 0x000000000534e25c 0x166200000534e25c 0x166bffffed8383fc
MERGE Data Link (2322): bad timestamp  6 3 0xc1a07f16 0x0070e9cc 0x000001e74070e9cc 0x166c01e74070e9cc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b37895 0x0070e9cc 0x000001e74070e9cc 0x166c01e74070e9cc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc19f7e41 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a27e22 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b47399 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ab7dad 0x0070f96c 0x000001e74070f96c 0x166c01e74070f96c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b564f9 0x0070f96c 0x000001e74070f96c 0x166c01e74070f96c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b60177 0x0071013c 0x000001e74071013c 0x166c01e74071013c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a57e95 0x00e2208c 0x000001e700e2208c 0x166c01e700e2208c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b81491 0x00e2208c 0x000001e700e2208c 0x166c01e700e2208c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ac7e4f 0x00e2285c 0x000001e700e2285c 0x166c01e700e2285c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b90000 0x00e2285c 0x000001e700e2285c 0x166c01e700e2285c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ba0c64 0x00e2302c 0x000001e700e2302c 0x166c01e700e2302c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bb6ea8 0x00e237fc 0x000001e700e237fc 0x166c01e700e237fc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a87dc6 0x00e23fcc 0x000001e700e23fcc 0x166c01e700e23fcc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bc0641 0x00e23fcc 0x000001e700e23fcc 0x166c01e700e23fcc 0x166c01e74071090c