To Synchronize the FEE64 ASIC clocks to rise at the same timestamp time use the System function Synchronise the ASIC clocks in the System Wide Checks browser page.

The Server will read the current timestamp value and calculate, based on the number of FEEs and the access delay, a timestamp value sometime in the future. F_stamp

F_stamp is written to each FEE and the synchronization is enabled. 

In a state machine in the FEE ASIC clock control, logic is enabled to synchronize its ASIC clock. It compares the current timestamp with F_stamp and if the current timestamp is >= F_stamp then it will start the ASIC clock.

The report in the browser log window gives the 3 LSBs of the timestamp at the instant the ASIC clock is started. They should all be the same. They are from the 10ns timestamp.

There is a new operation available in the System Functions menu in the System Wide Checks page. ( The page needs to be reset once when the FEEs are powered before this function will appear )

Check the Options files are all the same size

Using this function will provide a list of the file sizes of the Options/< fee name >/CONTENTS files along with the last time they were accessed as a list.

It's up to the user, at present, to interpret the results but more is possible if required ....

Here is a document showing how and where to calibrate the LMK3200 devices that lock to the system clock and generate the clocks for the ADCs and the FPGA internal logic.

We had issues with all TTYs reading out on nnrpi1, I have moved them from an apparently faulty USB hub to a new one.
Trying to reboot the pi to detect the new USB ports and it didn't recover from boot
Trying to repair SD card but I get a lot of errors on my laptop, indications are the SD card is completely dead
A new SD card will be ordered and nnrpi1 restored from the original images

A new USB hub should also be organised as it seems one is faulty (pi hangs if 3 or more USB devices connected to it)

In no particular order

1) Options

variables unexpectedly changing or becoming unset/undefined - see https://elog.ph.ed.ac.uk/DESPEC/205

Is this why *.L, *.H and Rate spectra are undefined in some FEE64s?


2) Merger 'Bad Timestamp' error reports - e.g. see https://elog.ph.ed.ac.uk/DESPEC/203

Meaning of report? Significance/impact?


3) Large numbers of PAUSE/RESUME for some (but not all) FEE64s - see e.g. https://elog.ph.ed.ac.uk/DESPEC/203

Significance/impact?

-- NH to add

4) Slow but non-zero increase of White Rabbit errors (mostly timestamp but much rarer ASIC errors)
We know the White Rabbit network in general is reliable but could we see interruption to the 200 MHz clock or 10 kHz timestamp when FATIMA DAQ is busy/restarting
Wasn't uniform over all FEEs implying a more local source though?

-- OH to add
5) The dropping of links between FEEs and the MERGER. Quite often these were correlated with the 'Bad Timestamp' error messages and also the WR errors

---- PJCS to add
6) The addition of the NIM clock-divider to allow better ASIC clock synchronisation.

Screenshots of successive displays of Options for FEE64s aida01, 02, 03 and 04.

12.40 The snout w/bPlast plus 3xDSSSDs was removed yesterday

      AIDA pulser tests via adaptor PCB test inputs, i.e. zero capacitative load and zero leakage current

      per FEE64 1.8.W spectra - attachments 1-4
      per FEE64 Rate spectra - attachment 5
      per FEE64 1.8.L spectra - attachments 6-7 

From the per FEE64 1.8.L

FEE64 FWHM

1  -
2  45
3  25
4  -
5  -
6  45
7  25
8  37
9  30
10 38
11 19
12 35

For comparison we would expect to obtain ~ 15-20 ch FWHM (~10.5-14 keV FWHM) based on tests
conducted at T9, STFC DL. The peak widths observed above are significantly worse than expected.
Are the adaptor PCBs currently grounded to FEE64 Cu mezzanine cover?

11.28 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

10.26 System wide checks
	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     25     16      3      6      1      3      2      2      3      4     10   : 54676
aida02 :     17     13     14     10      5      5      2      3      3      3      6   : 37516
aida03 :     19     10      8      3      2      3      2      4      2      3      6   : 36220
aida04 :      2      6      2      3      3      2      2      3      2      4     15   : 74360
aida05 :     89     48     20      3      3     34     21     20      6      4      7   : 64324
aida06 :     31     11      1     10      0      1      1      3      4      2      5   : 31140
aida07 :     21     13      8      4      3      2      4      4      3      3      6   : 37756
aida08 :     37     10      2      0      1      2      2      4      3      3      6   : 36932
aida09 :      4     18      6      3      1      3      1      2      3      3      6   : 35872
aida10 :     20      4      1      3      0      2      2      3      3      3      6   : 36320
aida11 :      2      6      6      2      2      4      2      3      2      3      6   : 35672
aida12 :     15     10      2      3      1      2      2      3      3      3      6   : 36428


      FEE64 Temperatures OK - attachment 1
      Good event, disc info #6 & good wave statistics - attachments 2, 3 & 4
           note high disc rates (c. 200k) in FEE64s aida03, aida05 & aida07
      Detector bias & leakage currents OK - attachment 5
      Merger OK - 4.5M data items/s
      TapeServer OK - 15Mb/s

      Rate spectra - attachment 6

11.41 Merger errors since yesterday's Elog


MERGE Data Link (20502): bad timestamp  2 3 0xc08e4cce 0x0c1c8460 0x0000f14c5c1c8460 0x166cf14c5c1c8460 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0967f4f 0x0c1c8460 0x0000f14c5c1c8460 0x166cf14c5c1c8460 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0807dc7 0x0c1c9bd0 0x0000f14c5c1c9bd0 0x166cf14c5c1c9bd0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0957ee9 0x0c1c9bd0 0x0000f14c5c1c9bd0 0x166cf14c5c1c9bd0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0857eea 0x0c1ca3a0 0x0000f14c5c1ca3a0 0x166cf14c5c1ca3a0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc0977df3 0x0c1ca3a0 0x0000f14c5c1ca3a0 0x166cf14c5c1ca3a0 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc08a7edd 0x0c1cab70 0x0000f14c5c1cab70 0x166cf14c5c1cab70 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7c7f 0x0c1cab70 0x0000f14c5c1cab70 0x166cf14c5c1cab70 0x166cf1515c1c74c0
MERGE Data Link (20502): bad timestamp  2 3 0xc09d7fd8 0x0c1cb340 0x0000f14c5c1cb340 0x166cf14c5c1cb340 0x166cf1515c1c74c0

 :
 :

MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c338 0x000018ff4433c338 0x166d18ff4433c338 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c356 0x000018ff4433c356 0x166d18ff4433c356 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263f080 0x0433c36a 0x000018ff4433c36a 0x166d18ff4433c36a 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x82630f7f 0x0433c374 0x000018ff4433c374 0x166d18ff4433c374 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c388 0x000018ff4433c388 0x166d18ff4433c388 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3a6 0x000018ff4433c3a6 0x166d18ff4433c3a6 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263efff 0x0433c3c4 0x000018ff4433c3c4 0x166d18ff4433c3c4 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3d8 0x000018ff4433c3d8 0x166d18ff4433c3d8 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263ffff 0x0433c3f6 0x000018ff4433c3f6 0x166d18ff4433c3f6 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x8263efff 0x0433c414 0x000018ff4433c414 0x166d18ff4433c414 0x166d18ff4e17c2c0
MERGE Data Link (20502): bad timestamp  2 3 0x82218ff4 0x0433c41e 0x000018ff4433c41e 0x166d18ff4433c41e 0x166d18ff4e17c2c0

From Tuesday, March 16, 2021 9:54:36.584 PM to Wednesday, March 17, 2021 10:02:05.935 AM

11.46 Grafana record of bias & leakage current for the 7 days to date - attachment 7

11.51 ASIC check load - all FEE64s, all ASICs

11.51 Disable 'no storage' mode and start writing to disk file S452/R58

12.01 R58_18 & R58_19 *no* change

12.16  R58_24 & R58_25 *all* waveforms disabled (Run Control tab) 

12.23 R58_27 & R58_28 *all* discriminators disabled (Discriminator tab) *but* disc rates now low - see attachment 8 cf. attachment 3

12.27 Enable 'no storage' mode

12.38 Analysis of files R58_19, R58_25 & R58_28 - see attachments 9, 10 & 11
      Disabling waveforms significantly reduces pause/resume for aida01 - aida08
      Effect of disabling discriminators less clear - disc rate zero for all FEE64s except aida03 (2k).

10.25 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

10.26 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

	
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


	
		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


	
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     15     19      5      5      1      3      2      2      3      4     10   : 54660
aida02 :     20     13     14     11      5      5      2      4      2      3      6   : 37048
aida03 :      2      3      7      3      2      3      2      4      2      3      6   : 36080
aida04 :     14     10      4      3      3      2      2      4      1      4     15   : 73960
aida05 :     97     51     19      3      3     34     21     20      6      4      7   : 64364
aida06 :     24      7      5      9      0      1      1      3      4      2      5   : 31112
aida07 :     21     13      9      3      3      2      3      4      3      3      6   : 37484
aida08 :     38     10      3      5      2      3      1      4      3      3      6   : 37048
aida09 :     24     11      4      3      2      3      2      3      3      3      6   : 36696
aida10 :     26     10      1      3      1      2      1      4      3      3      6   : 36712
aida11 :      0      7      7      1      1      4      2      3      2      3      6   : 35592
aida12 :      6      8      3      2      2      1      2      3      3      3      6   : 36296


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.5M data items/s
      TapeServer OK - 13Mb/s

      No merger server error messages since most recent check yesterday

15.50 Merger error messages


 :
 :

MERGE Data Link (20503): bad timestamp  3 3 0x838153cb 0x090bf960 0x0000c2c4f90bf960 0x166cc2c4f90bf960 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x090bf960 0x0000c2c4f90bf960 0x166cc2c4f90bf960 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d481de 0x090bff32 0x0000c2c4f90bff32 0x166cc2c4f90bff32 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c181f8 0x090c0ed2 0x0000c2c4f90c0ed2 0x166cc2c4f90c0ed2 0x166cc2c4fefc5fd2
MERGE Actor (20516): Working with 0 from 12 data sources

MERGE Data Link (20503): bad timestamp  3 3 0xc0c47f9e 0x090c16a2 0x0000c2c4f90c16a2 0x166cc2c4f90c16a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d08127 0x090c16a2 0x0000c2c4f90c16a2 0x166cc2c4f90c16a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c681ee 0x090c1e72 0x0000c2c4f90c1e72 0x166cc2c4f90c1e72 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d181c9 0x090c1e72 0x0000c2c4f90c1e72 0x166cc2c4f90c1e72 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98204 0x090c2642 0x0000c2c4f90c2642 0x166cc2c4f90c2642 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc8075 0x090c2e12 0x0000c2c4f90c2e12 0x166cc2c4f90c2e12 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cf812f 0x090c35e2 0x0000c2c4f90c35e2 0x166cc2c4f90c35e2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c780ef 0x090c4d52 0x0000c2c4f90c4d52 0x166cc2c4f90c4d52 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f7816c 0x090c8bd2 0x0000c2c4f90c8bd2 0x166cc2c4f90c8bd2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d480f1 0x090c93a2 0x0000c2c4f90c93a2 0x166cc2c4f90c93a2 0x166cc2c4fefc5fd2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c48039 0x090ca342 0x0000c2c4f90ca342 0x166cc2c4f90ca342 0x166cc2c4fefc5fd2

 :
 :

MERGE Data Link (20502): bad timestamp  2 3 0xc08d7cc1 0x085c7e10 0x0000c429685c7e10 0x166cc429685c7e10 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08e281c 0x085c85e0 0x0000c429685c85e0 0x166cc429685c85e0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7cc1 0x085c8db0 0x0000c429685c8db0 0x166cc429685c8db0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927e8c 0x085c9580 0x0000c429685c9580 0x166cc429685c9580 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08c414b 0x085ca520 0x0000c429685ca520 0x166cc429685ca520 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857dee 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987d2c 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07d90 0x085cbc90 0x0000c429685cbc90 0x166cc429685cbc90 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0877fd6 0x085d02e0 0x0000c429685d02e0 0x166cc429685d02e0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927e7c 0x085d2220 0x0000c429685d2220 0x166cc429685d2220 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857d7f 0x085d4930 0x0000c429685d4930 0x166cc429685d4930 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987cc0 0x085d4930 0x0000c429685d4930 0x166cc429685d4930 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07ccd 0x085d5100 0x0000c429685d5100 0x166cc429685d5100 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08e80fd 0x085d7fe0 0x0000c429685d7fe0 0x166cc429685d7fe0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0977e77 0x085d8f80 0x0000c429685d8f80 0x166cc429685d8f80 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0817ec8 0x085d9750 0x0000c429685d9750 0x166cc429685d9750 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0847fca 0x085d9f20 0x0000c429685d9f20 0x166cc429685d9f20 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc08c40e8 0x085da6f0 0x0000c429685da6f0 0x166cc429685da6f0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0927f3e 0x085db690 0x0000c429685db690 0x166cc429685db690 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0x82614000 0x085dbb36 0x0000c429685dbb36 0x166cc429685dbb36 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0857ebc 0x085ddda0 0x0000c429685ddda0 0x166cc429685ddda0 0x166cc429685df380
MERGE Data Link (20502): bad timestamp  2 3 0xc0987eec 0x085ddda0 0x0000c429685ddda0 0x166cc429685ddda0 0x166cc429685df380

21.04 Merger errors since 15.50

MERGE Data Link (20502): bad timestamp  2 3 0xc0907de6 0x03087130 0x0000ea6805fd8d80 0x166cea6803087130 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0878007 0x030880d0 0x0000ea68030880d0 0x166cea68030880d0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0937ec9 0x030888a0 0x0000ea68030888a0 0x166cea68030888a0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4e2d 0x0308a7e0 0x0000ea680308a7e0 0x166cea680308a7e0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0998001 0x0308bf50 0x0000ea680308bf50 0x166cea680308bf50 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4abc 0x0308fdd0 0x0000ea680308fdd0 0x166cea680308fdd0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0907fb1 0x030905a0 0x0000ea68030905a0 0x166cea68030905a0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0937fd8 0x03091d10 0x0000ea6803091d10 0x166cea6803091d10 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0807df1 0x03093480 0x0000ea6803093480 0x166cea6803093480 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d14 0x03093c50 0x0000ea6803093c50 0x166cea6803093c50 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x8280dbb0 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x82816a12 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x03093e26 0x0000ea6803093e26 0x166cea6803093e26 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e07 0x030953c0 0x0000ea68030953c0 0x166cea68030953c0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0967c2a 0x030953c0 0x0000ea68030953c0 0x166cea68030953c0 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0887d0f 0x03095b90 0x0000ea6803095b90 0x166cea6803095b90 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0997dad 0x03095b90 0x0000ea6803095b90 0x166cea6803095b90 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7bfc 0x03096360 0x0000ea6803096360 0x166cea6803096360 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7c98 0x03096360 0x0000ea6803096360 0x166cea6803096360 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc09d7f7e 0x03096b30 0x0000ea6803096b30 0x166cea6803096b30 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e10 0x03098a70 0x0000ea6803098a70 0x166cea6803098a70 0x166cea6805fd8d80
MERGE Data Link (20502): bad timestamp  2 3 0xc0957f91 0x048638a0 0x0000ea6bf48638a0 0x166cea6bf48638a0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967e3d 0x04864070 0x0000ea6bf4864070 0x166cea6bf4864070 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4dd3 0x04865fb0 0x0000ea6bf4865fb0 0x166cea6bf4865fb0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0998027 0x04866780 0x0000ea6bf4866780 0x166cea6bf4866780 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7dd9 0x0486a600 0x0000ea6bf486a600 0x166cea6bf486a600 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d45 0x0486add0 0x0000ea6bf486add0 0x166cea6bf486add0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7e98 0x0486b5a0 0x0000ea6bf486b5a0 0x166cea6bf486b5a0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967d19 0x0486cd10 0x0000ea6bf486cd10 0x166cea6bf486cd10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e95 0x0486ec50 0x0000ea6bf486ec50 0x166cea6bf486ec50 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0997dba 0x0486f420 0x0000ea6bf486f420 0x166cea6bf486f420 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x8280cc32 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82816a15 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x04871a90 0x0000ea6bf4871a90 0x166cea6bf4871a90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4c2e 0x04874240 0x0000ea6bf4874240 0x166cea6bf4874240 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7ea0 0x04874a10 0x0000ea6bf4874a10 0x166cea6bf4874a10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4b7e 0x048780c0 0x0000ea6bf48780c0 0x166cea6bf48780c0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0857dd8 0x0487a7d0 0x0000ea6bf487a7d0 0x166cea6bf487a7d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947ed6 0x0487a7d0 0x0000ea6bf487a7d0 0x166cea6bf487a7d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7f9f 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0967f8e 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07d73 0x0487afa0 0x0000ea6bf487afa0 0x166cea6bf487afa0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977d81 0x0487b770 0x0000ea6bf487b770 0x166cea6bf487b770 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7d98 0x0487bf40 0x0000ea6bf487bf40 0x166cea6bf487bf40 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09d803b 0x0487c710 0x0000ea6bf487c710 0x166cea6bf487c710 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4ca6 0x0487d6b0 0x0000ea6bf487d6b0 0x166cea6bf487d6b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x8280cc32 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82816a15 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x04880a9a 0x0000ea6bf4880a9a 0x166cea6bf4880a9a 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27dba 0x04880d60 0x0000ea6bf4880d60 0x166cea6bf4880d60 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4996 0x04881d00 0x0000ea6bf4881d00 0x166cea6bf4881d00 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0857f80 0x04883470 0x0000ea6bf4883470 0x166cea6bf4883470 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f50 0x04883c40 0x0000ea6bf4883c40 0x166cea6bf4883c40 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07e3b 0x04884410 0x0000ea6bf4884410 0x166cea6bf4884410 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977f31 0x048853b0 0x0000ea6bf48853b0 0x166cea6bf48853b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7db2 0x04885b80 0x0000ea6bf4885b80 0x166cea6bf4885b80 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4951 0x04886b20 0x0000ea6bf4886b20 0x166cea6bf4886b20 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27e54 0x0488a1d0 0x0000ea6bf488a1d0 0x166cea6bf488a1d0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4bc5 0x0488b170 0x0000ea6bf488b170 0x166cea6bf488b170 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0947d43 0x0488d0b0 0x0000ea6bf488d0b0 0x166cea6bf488d0b0 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0877f7a 0x0488e820 0x0000ea6bf488e820 0x166cea6bf488e820 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0977e13 0x0488e820 0x0000ea6bf488e820 0x166cea6bf488e820 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4985 0x0488ff90 0x0000ea6bf488ff90 0x166cea6bf488ff90 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08b7f0e 0x04893640 0x0000ea6bf4893640 0x166cea6bf4893640 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc0b27dc7 0x04893640 0x0000ea6bf4893640 0x166cea6bf4893640 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4db1 0x04893e10 0x0000ea6bf4893e10 0x166cea6bf4893e10 0x166cea6bfa7c5dac
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4ea2 0x07d8b160 0x0000ea6d67d8b160 0x166cea6d67d8b160 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7eda 0x07d8b930 0x0000ea6d67d8b930 0x166cea6d67d8b930 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09b7c0e 0x07d8d0a0 0x0000ea6d67d8d0a0 0x166cea6d67d8d0a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4bdb 0x07d8efe0 0x0000ea6d67d8efe0 0x166cea6d67d8efe0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947de3 0x07d90750 0x0000ea6d67d90750 0x166cea6d67d90750 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0877fed 0x07d945d0 0x0000ea6d67d945d0 0x166cea6d67d945d0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4f9a 0x07d94da0 0x0000ea6d67d94da0 0x166cea6d67d94da0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07f5e 0x07d94da0 0x0000ea6d67d94da0 0x166cea6d67d94da0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08d7e2b 0x07d95570 0x0000ea6d67d95570 0x166cea6d67d95570 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4c6a 0x07d98c20 0x0000ea6d67d98c20 0x166cea6d67d98c20 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f27 0x07d99bc0 0x0000ea6d67d99bc0 0x166cea6d67d99bc0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7d09 0x07d9a390 0x0000ea6d67d9a390 0x166cea6d67d9a390 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937d1a 0x07d9bb00 0x0000ea6d67d9bb00 0x166cea6d67d9bb00 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x8280e345 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82816a16 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x07d9bd08 0x0000ea6d67d9bd08 0x166cea6d67d9bd08 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0878187 0x07d9e210 0x0000ea6d67d9e210 0x166cea6d67d9e210 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07ce8 0x07d9e210 0x0000ea6d67d9e210 0x166cea6d67d9e210 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c38a0 0x07d9e9e0 0x0000ea6d67d9e9e0 0x166cea6d67d9e9e0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4b25 0x07da2090 0x0000ea6d67da2090 0x166cea6d67da2090 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7c55 0x07da3030 0x0000ea6d67da3030 0x166cea6d67da3030 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937e6c 0x07da47a0 0x0000ea6d67da47a0 0x166cea6d67da47a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c36aa 0x07da7680 0x0000ea6d67da7680 0x166cea6d67da7680 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07c8e 0x07da7680 0x0000ea6d67da7680 0x166cea6d67da7680 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837f38 0x07da9d90 0x0000ea6d67da9d90 0x166cea6d67da9d90 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0917d84 0x07daad30 0x0000ea6d67daad30 0x166cea6d67daad30 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0947f39 0x07dab500 0x0000ea6d67dab500 0x166cea6d67dab500 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0977d15 0x07dabcd0 0x0000ea6d67dabcd0 0x166cea6d67dabcd0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d6c 0x07dac4a0 0x0000ea6d67dac4a0 0x166cea6d67dac4a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7f77 0x07dac4a0 0x0000ea6d67dac4a0 0x166cea6d67dac4a0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x8280e346 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82816a16 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0x82820000 0x07dad832 0x0000ea6d67dad832 0x166cea6d67dad832 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937f84 0x07dadc10 0x0000ea6d67dadc10 0x166cea6d67dadc10 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4bb4 0x07db0af0 0x0000ea6d67db0af0 0x166cea6d67db0af0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07cee 0x07db0af0 0x0000ea6d67db0af0 0x166cea6d67db0af0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e94 0x07db3200 0x0000ea6d67db3200 0x166cea6d67db3200 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4d16 0x07db5910 0x0000ea6d67db5910 0x166cea6d67db5910 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0937dd1 0x07db7080 0x0000ea6d67db7080 0x166cea6d67db7080 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7e42 0x07db87f0 0x0000ea6d67db87f0 0x166cea6d67db87f0 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4ed0 0x07db9f60 0x0000ea6d67db9f60 0x166cea6d67db9f60 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0a07e15 0x07db9f60 0x0000ea6d67db9f60 0x166cea6d67db9f60 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0977f69 0x07dbaf00 0x0000ea6d67dbaf00 0x166cea6d67dbaf00 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837e60 0x07dbc670 0x0000ea6d67dbc670 0x166cea6d67dbc670 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4bfb 0x07dbed80 0x0000ea6d67dbed80 0x166cea6d67dbed80 0x166cea6d6938be60
MERGE Data Link (20502): bad timestamp  2 3 0xc0837eb4 0x055a8c50 0x0000ea6d79a8c844 0x166cea6d755a8c50 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7dba 0x055a8c50 0x0000ea6d755a8c50 0x166cea6d755a8c50 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e6d 0x055a9420 0x0000ea6d755a9420 0x166cea6d755a9420 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc0907bef 0x055aa3c0 0x0000ea6d755aa3c0 0x166cea6d755aa3c0 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08b7ed5 0x055acad0 0x0000ea6d755acad0 0x166cea6d755acad0 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc0887eb6 0x055ae240 0x0000ea6d755ae240 0x166cea6d755ae240 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08e4da2 0x055aea10 0x0000ea6d755aea10 0x166cea6d755aea10 0x166cea6d79a8c844
MERGE Data Link (20502): bad timestamp  2 3 0xc08f7ff1 0x055b0180 0x0000ea6d755b0180 0x166cea6d755b0180 0x166cea6d79a8c844

 :
 :

MERGE Data Link (20503): bad timestamp  3 3 0xc0da81a0 0x0bb28852 0x0000ee992bb28852 0x166cee992bb28852 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f4811f 0x0bb29022 0x0000ee992bb29022 0x166cee992bb29022 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc805b 0x0bb297f2 0x0000ee992bb297f2 0x166cee992bb297f2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0d38174 0x0bb2af62 0x0000ee992bb2af62 0x166cee992bb2af62 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0c38233 0x0bb2ede2 0x0000ee992bb2ede2 0x166cee992bb2ede2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0xc0f68019 0x0bb2f5b2 0x0000ee992bb2f5b2 0x166cee992bb2f5b2 0x166cee992d6fc8b2
MERGE Data Link (20503): bad timestamp  3 3 0x83807863 0x0bb30fca 0x0000ee992bb30fca 0x166cee992bb30fca 0x166cee992d6fc8b2
MERGE Data Link (20502): bad timestamp  2 3 0xc0877f3c 0x0e20b4a0 0x0000eeae2e20b4a0 0x166ceeae2e20b4a0 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc09a7bc6 0x0e20b4a0 0x0000eeae2e20b4a0 0x166ceeae2e20b4a0 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc08c4e12 0x0e20bc70 0x0000eeae2e20bc70 0x166ceeae2e20bc70 0x166ceeae40e4c770
MERGE Data Link (20502): bad timestamp  2 3 0xc09c7df4 0x0e20bc70 0x0000eeae2e20bc70 0x166ceeae2e20bc70 0x166ceeae40e4c770

Earliest & latest timestamps from latest batch = Tuesday, March 16, 2021 7:48:18.767 PM - Tuesday, March 16, 2021 9:06:38.184 PM this evening
 
21.17 FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.5M data items/s
      TapeServer OK - 14Mb/s

12.48 DAQ continues OK - no storage

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

12.50 System wide checks

	
FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module



		 Base 		Current 	Difference
aida05 fault 	 0x6acb : 	 0x6acc : 	 1  
aida06 fault 	 0x8394 : 	 0x8395 : 	 1  
aida07 fault 	 0x8c77 : 	 0x8c7a : 	 3  
aida08 fault 	 0x1692 : 	 0x1693 : 	 1  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR



FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     20     18      5      4      2      2      2      2      3      4     10   : 54576
aida02 :     26     18     16     10      6      4      2      4      2      3      6   : 37048
aida03 :     18     10      4      3      2      3      2      4      2      3      6   : 36152
aida04 :     13      9      4      2      2      2      2      4      1      4     15   : 73852
aida05 :     46     72     70     54     52     48     21     19      7      4      7   : 72216
aida06 :     32     10      4     10      0      2      2      4      3      2      5   : 31056
aida07 :     19      7      5      3      2      3      3      3      3      3      6   : 36916
aida08 :     20     14      2      3      1      3      1      4      3      3      6   : 36864
aida09 :      3      6      3      3      2      4      1      2      3      3      6   : 35916
aida10 :     23     11      3      3      1      3      3      2      3      3      6   : 36356
aida11 :      2      5      6      2      2      4      2      3      2      3      6   : 35664
aida12 :     25     10      5      2      2      2      2      3      3      3      6   : 36548


      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 14Mb/s

12.55 Rate spectra - attachments 4 & 5

      Merger server error messages since most recent check

MERGE Data Link (20503): bad timestamp  3 3 0xc0d582ee 0x04f02352 0x000064aa97efd660 0x166c64aa94f02352 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e680cf 0x04f032f2 0x000064aa94f032f2 0x166c64aa94f032f2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c68215 0x04f03ac2 0x000064aa94f03ac2 0x166c64aa94f03ac2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd81f1 0x04f061d2 0x000064aa94f061d2 0x166c64aa94f061d2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0eb812a 0x04f07942 0x000064aa94f07942 0x166c64aa94f07942 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0f68041 0x04f07942 0x000064aa94f07942 0x166c64aa94f07942 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83806b78 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83811ea0 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x04f07aa0 0x000064aa94f07aa0 0x166c64aa94f07aa0 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c183d5 0x04f0aff2 0x000064aa94f0aff2 0x166c64aa94f0aff2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d58220 0x04f0b7c2 0x000064aa94f0b7c2 0x166c64aa94f0b7c2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e68097 0x04f0bf92 0x000064aa94f0bf92 0x166c64aa94f0bf92 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c682e7 0x04f0c762 0x000064aa94f0c762 0x166c64aa94f0c762 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cf7ff7 0x04f0cf32 0x000064aa94f0cf32 0x166c64aa94f0cf32 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd817e 0x04f0ee72 0x000064aa94f0ee72 0x166c64aa94f0ee72 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e180e5 0x04f105e2 0x000064aa94f105e2 0x166c64aa94f105e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d181fd 0x04f10db2 0x000064aa94f10db2 0x166c64aa94f10db2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d78072 0x04f11582 0x000064aa94f11582 0x166c64aa94f11582 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c17fff 0x04f14462 0x000064aa94f14462 0x166c64aa94f14462 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c68234 0x04f15bd2 0x000064aa94f15bd2 0x166c64aa94f15bd2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c780f9 0x04f17342 0x000064aa94f17342 0x166c64aa94f17342 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98277 0x04f17b12 0x000064aa94f17b12 0x166c64aa94f17b12 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d28174 0x04f17b12 0x000064aa94f17b12 0x166c64aa94f17b12 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d5812d 0x04f182e2 0x000064aa94f182e2 0x166c64aa94f182e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0fd8112 0x04f182e2 0x000064aa94f182e2 0x166c64aa94f182e2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83806b78 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83811ea0 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0x83820000 0x04f186a2 0x000064aa94f186a2 0x166c64aa94f186a2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d98224 0x04f18ab2 0x000064aa94f18ab2 0x166c64aa94f18ab2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e68096 0x04f19a52 0x000064aa94f19a52 0x166c64aa94f19a52 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c080e1 0x04f1a222 0x000064aa94f1a222 0x166c64aa94f1a222 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d18282 0x04f1a222 0x000064aa94f1a222 0x166c64aa94f1a222 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c38198 0x04f1a9f2 0x000064aa94f1a9f2 0x166c64aa94f1a9f2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cb8187 0x04f1b1c2 0x000064aa94f1b1c2 0x166c64aa94f1b1c2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0cc8178 0x04f1b992 0x000064aa94f1b992 0x166c64aa94f1b992 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c78063 0x04f1ffe2 0x000064aa94f1ffe2 0x166c64aa94f1ffe2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0e980e1 0x04f1ffe2 0x000064aa94f1ffe2 0x166c64aa94f1ffe2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0c98129 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0d28107 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660
MERGE Data Link (20503): bad timestamp  3 3 0xc0eb80e2 0x04f207b2 0x000064aa94f207b2 0x166c64aa94f207b2 0x166c64aa97efd660

18.47 FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 14Mb/s

      No new merger error messages since 12.55 today

          

18.01 Pulser walkthrough at end of run S452 file S452/R57_0 ... R57_28

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

      BNC PB-5 settings changed by terminal command line (see attachments 1, 2 & 3)
      using the commands

      set rep rate 25
      set attenuation 5
      set amplitude 10
      set amplitude 9
      :
      :
      set amplitude 1
      set attenuation 1
      set rep rate 2

      Example n+n LEC spectrum - see attachment 4

18.27 R57 ends OK file R57_28

> 08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151
> 
> Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
> It is also the least noisy FEE in the back DSSD
> 
> MERGE Data Link (20510): bad timestamp  10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
> MERGE Data Link (20510): bad timestamp  10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262
> 
> 09:01 System wide checks
> 
> 		 Base 		Current 	Difference
> aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
> White Rabbit error counter test result: Passed 11, Failed 1
> 
> Understand the status reports as follows:-
> Status bit 3 : White Rabbit decoder detected an error in the received data
> Status bit 2 : Firmware registered WR error, no reload of Timestamp
> Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
> 
> 			 Base 		Current 		Difference
> aida07 fault 	 0x2 : 	 0x5 : 	 3  
> FPGA Timestamp error counter test result: Passed 11, Failed 1
> If any of these counts are reported as in error
> The ASIC readout system has detected a timeslip.
> That is the timestamp read from the time FIFO is not younger than the last
> 
>      Statistics - attachment 1
>      Temperature - attachment 2
>      Bias and leakage current - attachment 3
It might be worth considering decreasing the time between the flushing of slower FEEs. Is the Merger waiting for the slower data and causing ‘deadtime’ in the faster ones. Perhaps consider this as a 
balancing across the system? 

08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151

Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
It is also the least noisy FEE in the back DSSD

MERGE Data Link (20510): bad timestamp  10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
MERGE Data Link (20510): bad timestamp  10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262

09:01 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

     Statistics - attachment 1
     Temperature - attachment 2
     Bias and leakage current - attachment 3

11:18 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c78 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

     Statistics - attachment 4
     Temperatures - attachment 5
     Bias and leakage - attachment 6


13.14 DAQ continues OK - file R56_285

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

13.18 System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c78 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     27      8      3      4      2      2      2      2      3      4     10   : 54492
aida02 :      0      2      3      0      2      3      2      3      3      3      6   : 36416
aida03 :     18      9      1      2      0      4      1      3      3      3      6   : 36320
aida04 :      2      3      3      3      1      4      3      2      2      4     15   : 74224
aida05 :     26     72     72     54     53     49     22     18      7      4      7   : 72104
aida06 :      7      5      5     10      0      2      2      4      3      2      5   : 30932
aida07 :      2      4      2      1      1      3      2      3      2      3      7   : 39464
aida08 :     25      9      4      2      1      2      1      4      3      3      6   : 36716
aida09 :      4     11      3      5      2      4      1      2      3      3      6   : 36024
aida10 :     18      5      2      1      0      5      2      2      3      3      6   : 36144
aida11 :     25     10      4      1      1      4      2      3      2      3      6   : 35668
aida12 :     26      7      1      1      1      3      2      3      3      3      6   : 36496

13.19 FEE64 temperatures OK - attachment 7
      Good events stats OK - attachment 8
      Bias & leakage currents OK - attachment 9
      Merger 4.5M items/s
      Tape Server 14Mb/s
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      All histograms zero'd


16.00 DAQ contiues R56_361

       System wide checks

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	 Base 		Current 	Difference
aida07 fault 	 0x8c77 : 	 0x8c79 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     23      8      6      4      2      2      2      2      3      4     10   : 54524
aida02 :      1      3      4      2      1      2      1      4      3      3      6   : 36572
aida03 :      2      3      2      3      1      3      2      2      3      3      6   : 35936
aida04 :     19      7      2      3      1      4      3      3      1      4     15   : 73796
aida05 :     34     68     72     54     53     49     22     18      7      4      7   : 72104
aida06 :     11     10      4     10      0      2      2      4      3      2      5   : 30972
aida07 :      1      3      2      2      2      2      2      3      2      3      7   : 39420
aida08 :     24     10      5      2      1      3      1      4      3      3      6   : 36864
aida09 :     20      8      3      5      2      3      2      3      3      3      6   : 36704
aida10 :      2      3      2      3      1      3      3      2      3      3      6   : 36192
aida11 :     17     10      4      0      1      4      2      3      2      3      6   : 35604
aida12 :     20      6      3      2      1      3      2      2      2      4      6   : 37040


15.05 FEE64 temperatures OK - attachment 10
      Good events stats OK - attachment 11
      Bias & leakage currents OK - attachment 12
      Merger 4.6M items/s
      Tape Server 14Mb/s
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      Rate spectra - attachments 13 & 14
      p+n junction HEC spectra - attachment 15

17.33 end of S452 run file R56_397
      No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.

      BNC PB-5 config - attachment 16
      CAEN 8xxx NIM bin config - attachment 17

00:08 DAQ continues ok - writing to file R46_221

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

00:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a4 : 	 4  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     11      6      2      5      1      3      2      2      3      4     10   : 54492
aida02 :     12      7      3      4      3      3      3      2      3      3      6   : 36440
aida03 :      4      4      2      4      2      3      2      2      3      3      6   : 36048
aida04 :     18      9      4      2      1      3      2      4      1      4     15   : 73936
aida05 :     31     11      8      7      6      4      2     18      7      4      7   : 55220
aida06 :     36      8     18      4      4      2      1      4      3      2      5   : 31088
aida07 :     26      7      4      3      1      2      1      3      3      3      6   : 36224
aida08 :     25     12      5      4      4      3      2      3      3      3      6   : 36884
aida09 :      2      9      4      2      2      2      2      3      3      3      6   : 36432
aida10 :     11      3      4      6      4      3      1      3      3      3      6   : 36548
aida11 :     21      8      5      0      1      3      3      3      2      3      6   : 35748
aida12 :     10      6      6      2      4      4      1      3      3      3      6   : 36600

00:14 FEE64 Temperatures ok - attachment 1
      Good event statistics ok - attachment 2
      Detector bias/leakage currents ok - attachment 3

00:17 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

01:30 beam off

01:31 beam back

02:19 system wide checks

      same as entry at 00:09, *except*

 		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a5 : 	 5  
White Rabbit error counter test result: Passed 11, Failed 1

02:22 FEE64 Temperatures ok - attachment 4
      Good event statistics ok - attachment 5
      Detector bias/leakage currents ok - attachment 6

02:30 Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s

      no further bad timestamp errors in NewMerger terminal

02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7

02:50 ... and sure enough lose connection to aida07 after checking stats

      DAQ stop, Merger/TapeServer closed

      telnet aida07 -> sent reboot command

      mounted ok, reset DAQ, TapeService and Merge

      restarted DATA forwarding to MBS

      all looks to be ok now -> writing to R56

03:09 system wide checks:

FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

	        Base 	Current Difference
aida07 fault 	 0x2 : 	 0x5 : 	 3  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     13      5      3      7      2      2      2      2      3      4     10   : 54508
aida02 :      2      7      3      2      2      3      1      4      3      3      6   : 36784
aida03 :      8      8      1      3      2      3      1      3      3      3      6   : 36304
aida04 :     20      9      2      3      0      3      2      4      1      4     15   : 73880
aida05 :     51     74     70     54     53     49     22     18      7      4      7   : 72188
aida06 :     15     15      3      9      1      2      2      4      3      2      5   : 31044
aida07 :     24      6      2      3      3      4      2      3      2      3      7   : 39888
aida08 :     19     15      1      2      1      3      1      4      3      3      6   : 36820
aida09 :     18      4      2      7      3      4      1      3      3      3      6   : 36648
aida10 :     20      7      1      1      1      3      3      2      3      3      6   : 36216
aida11 :     25      7      5      3      4      2      1      4      2      3      6   : 35916
aida12 :      3      3      2      2      3      2      3      2      3      3      6   : 36164

03:18 FEE64 Temperatures ok - attachment 8
      Good event statistics ok - attachment 9
      Detector bias/leakage currents ok - attachment 10

      Merger ok - 4.4M items/s
      TapeServer ok - 14 MB/s
      
03:29 more bad timestamp error messages - attachment 11

03:52 analyser output on R56_35 - attachment 12

      no timewarps

03:53 DAQ continues ok, writing to file R56_41

05:18 system wide checks:

      same as entry at 03:08, *but* with additional WR decoder status error

Base 		Current 	Difference
aida07 fault 	 0x8c74 : 	 0x8c77 : 	 3  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

05:19 FEE64 Temperatures ok - attachment 14
      Good event statistics ok - attachment 13
      Detector bias/leakage currents ok - attachment 15

      Merger ok - 4.4M items/s
      TapeServer ok - 15 MB/s

05:25 rate spectra - attachment 16

05:33 AIDA implant XY hit patterns - attachments 17/18/19

05:56 BuTIS interface control set to 0x3 at 04.52 UTC

07:06 system wide checks - same error messages as entry at 05:18

07:08 FEE64 Temperatures ok - attachment 20

      good event statistics ok - attachment 21

      detector bias/leakage currents ok - attachment 22

      Merger ok - 4.3M items/s
      TapeServer ok - 15 MB/s

07:22 no bad timestamp errors since 02:30 UTC

18.00 DAQ continues OK - file R46_64

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

18.02 System wide checks OK *except*

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.6M data items/s
      TapeServer OK - 16Mb/s

      All histograms zero'd

17.15 Most recent messages in merger server terminal session

MERGE Data Link (2322): bad timestamp  6 3 0xc1b57e35 0x0ba1685c 0x0000ef3fbba1685c 0x166bef3fbba1685c 0x166bef3fbba1879c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a77dc2 0x0ba1702c 0x0000ef3fbba1702c 0x166bef3fbba1702c 0x166bef3fbba1879c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bf7f66 0x0ba1702c 0x0000ef3fbba1702c 0x166bef3fbba1702c 0x166bef3fbba1879c
Transfer Error - : Broken pipe
Error 32
1: send() failed:
TCP transfer library version 4.0
1: TCP socket send buffer was 16384 - now 249856
1: TCP socket receive buffer was 87380 - now 249856
1: TCP socket created OK - now connecting to localhost port 10305
1: Connected to localhost port 10305


22.05 DAQ continues OK - file R46_168

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable

FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0x82a0 : 	 0x82a2 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x2 : 	 0x3 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :     18      6      7      5      1      3      1      3      3      4     10   : 54856
aida02 :     19      6      4      3      4      4      2      3      3      3      6   : 36892
aida03 :     22      8      1      2      1      2      1      3      3      3      6   : 36136
aida04 :     24      8      4      2      1      3      2      4      1      4     15   : 73952
aida05 :     37     14      9      8      7      3      2     18      7      4      7   : 55252
aida06 :     28     12     16      4      4      2      1      4      3      2      5   : 31056
aida07 :     28      9      2      2      2      2      1      3      3      3      6   : 36248
aida08 :     21     12      6      4      3      4      2      3      3      3      6   : 36948
aida09 :     18      5      2      1      1      3      2      2      3      3      6   : 35952
aida10 :     17      8      4      5      3      3      1      3      3      3      6   : 36516
aida11 :     21      8      5      0      0      4      3      3      2      3      6   : 35812
aida12 :     13     11      3      3    


      FEE64 Temperatures OK - attachment 4
      Good event statistics OK - attachment 5
      Detector bias & leakage currents OK - attachment 6
      Merger OK - 4.4M data items/s
      TapeServer OK - 15Mb/s

22.17 Rate spectra - attachments 7 & 8
      p+n junction HEC spectra - attachment 9

      Merger server error messages since 17.15

MERGE Data Link (2316): bad timestamp  0 3 0x80300000 0x0164e7f8 0x000000000164e7f8 0x166200000164e7f8 0x166bffffffa7654e
MERGE Data Link (2324): bad timestamp  8 3 0x88300000 0x0534e25c 0x000000000534e25c 0x166200000534e25c 0x166bffffed8383fc
MERGE Data Link (2322): bad timestamp  6 3 0xc1a07f16 0x0070e9cc 0x000001e74070e9cc 0x166c01e74070e9cc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b37895 0x0070e9cc 0x000001e74070e9cc 0x166c01e74070e9cc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc19f7e41 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a27e22 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b47399 0x0070f19c 0x000001e74070f19c 0x166c01e74070f19c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ab7dad 0x0070f96c 0x000001e74070f96c 0x166c01e74070f96c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b564f9 0x0070f96c 0x000001e74070f96c 0x166c01e74070f96c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b60177 0x0071013c 0x000001e74071013c 0x166c01e74071013c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a57e95 0x00e2208c 0x000001e700e2208c 0x166c01e700e2208c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b81491 0x00e2208c 0x000001e700e2208c 0x166c01e700e2208c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ac7e4f 0x00e2285c 0x000001e700e2285c 0x166c01e700e2285c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1b90000 0x00e2285c 0x000001e700e2285c 0x166c01e700e2285c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1ba0c64 0x00e2302c 0x000001e700e2302c 0x166c01e700e2302c 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bb6ea8 0x00e237fc 0x000001e700e237fc 0x166c01e700e237fc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1a87dc6 0x00e23fcc 0x000001e700e23fcc 0x166c01e700e23fcc 0x166c01e74071090c
MERGE Data Link (2322): bad timestamp  6 3 0xc1bc0641 0x00e23fcc 0x000001e700e23fcc 0x166c01e700e23fcc 0x166c01e74071090c

ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse


08:09 System wide checks ok *except*

      aida09 fails clock check

      aida09 calibration failed

08:14 FEE64 temperatures ok - attachment 1

      good event statistics ok - attachment 2

      detector bias/leakage current ok - attachment 3

08:22 Merger 4.3M items/s

      Tapeserver 14 MB/s

08:24 DAQ ok, writing to file R43_45

08:39 rate spectra - attachment 4

      aida09 spectrum not showing, however continues to collect statistics ok

10:03 System wide checks ok *except*

      aida09 fails clock check

      aida09 calibration failed

      FEE64 temperatures ok - attachment 5

      good event statistics ok - attachment 6

      detector bias/leakage current ok - attachment 7

10:10 Merger 4.5M items/s

      Tapeserver 14 MB/s

      data forwarding to MBS ok

10:11 writing to file R43_91

12.00 (LS)
      System wide checks:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      		         Base 		Current 	Difference
      aida07 fault 	 0x829e : 	 0x829f : 	 1  
      White Rabbit error counter test result: Passed 11, Failed 1
      Understand the status reports as follows:-
      Status bit 3 : White Rabbit decoder detected an error in the received data
      Status bit 2 : Firmware registered WR error, no reload of Timestamp
      Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

      no FGPA timestamp errors all passed

      Statistics (attachment8)
      Spectra rate (attachment9), aida09 still does not show histogram, but on run control are enabled for all fees. still 
      showing good events and other information (temperatures etc)
      FEE temps (attachment10)
      Leakage currents written to sheets (attachment11)
      Merger~4.6M items/s
      Tapeserver~14MB/s

12.36 bunch of bad timestamp errors in the new merger terminal (attachment 12) around file R43_152
      this error is still occurring at 12.45 will continue to monitor, all other system checks are reporting as normal


13.50 beam down file R43_183
      beam back a couple of minutes later

14.00 System wide checks:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      no white rabbit or FPGA errors all passed

      Statistics (attachment13)
      Spectra rate (attachment14)
      FEE temps (attachment15)
      Leakage currents written to sheets (attachment16)
      Merger~4.6M items/s
      Tapeserver~14MB/s  
      Writing to MBS okay, still seeing bad timestamp errors in the new merger terminal continuing to monitor and update 
      statistics tab every 20 to 30 minutes 

14.12 Analysed files R43_185,186,187 from timestamp error in ucesb, but see no timewarps  

14.30 have not seen any bad timestamp errors in new merger terminal for a while, unsure if something has been changed which 
      has stopped them

15.25 beam down
15.53 beam back

16.00 moved back to writing to /media/SecondDrive which should last until the end of the experiment
      now writing R46

16.10 System wide checks okay except:
      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      
      FEE64 module aida09 failed
      Calibration test result: Passed 11, Failed 1
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment17)
      Spectra rate (attachment18)
      FEE temps (attachment19)
      Leakage currents written to sheets (attachment20)
      Merger~4.7M items/s
      Tapeserver~14MB/s  
      Writing to MBS okay

24:00 Joined in with TD to debug the dropout of AIDA05 - See https://elog.ph.ed.ac.uk/DESPEC/193
      After both powercycles and telnet reboots we were still seeing 0 in the statistics but did note that the counter was > 0
      The last check we did was a restart of the merger server. This solved the issue.
      It is likely that the link to the FEE was dropped but not re-established upon the resets.
      There was no error message that this was seen. From now on I recommend refreshing the statistics ever 30 minutes.

01:29 System wide check
N.B. May not have reset baselines after reset
WR fault
		 Base 		Current 	Difference
aida01 fault 	 0x1ad : 	 0x1af : 	 2  
aida02 fault 	 0x4434 : 	 0x4436 : 	 2  
aida03 fault 	 0xcf95 : 	 0xcf99 : 	 4  
aida04 fault 	 0x25aa : 	 0x25ae : 	 4  
aida05 fault 	 0x4b4 : 	 0x4b7 : 	 3  
aida06 fault 	 0x834 : 	 0x837 : 	 3  
aida07 fault 	 0x146d : 	 0x1472 : 	 5  
aida08 fault 	 0xf4a8 : 	 0xf4ab : 	 3  
aida09 fault 	 0x4f6f : 	 0x4f73 : 	 4  
aida10 fault 	 0x7504 : 	 0x7506 : 	 2  
aida11 fault 	 0x26fa : 	 0x26fc : 	 2  
aida12 fault 	 0x2858 : 	 0x285b : 	 3  
White Rabbit error counter test result: Passed 0, Failed 12

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA Check
			 Base 		Current 		Difference
aida12 fault 	 0x0 : 	 0x4 : 	 4  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

01:42 Statistics - attachment 1
      Temperature - attachment 2
      Bias and leakage current - attachment 3

03:57 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x1472 : 	 0x1473 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida12 fault 	 0x4 : 	 0xc : 	 8  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

      Statistics - attachment 4
      Temp - attachment 5
      Bias - attachment 6

05:27 System wide checks

		 Base 		Current 	Difference
aida07 fault 	 0x1472 : 	 0x1474 : 	 2  
White Rabbit error counter test result: Passed 11, Failed 1

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

			 Base 		Current 		Difference
aida07 fault 	 0x0 : 	 0x1 : 	 1  
aida12 fault 	 0x4 : 	 0x16 : 	 18  
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last


     Statistics - attachment 7
     Temp - attachment 8
     Bias - attachment 9

05:51 Looking into the bad timestamp messages in the merger e.g.
MERGE Data Link (28348): bad timestamp  5 3 0xc14f8415 0x00eea280 0x0000cd2e70eea280 0x166bcd2e70eea280 0x166bcd2e716c1f80

Looking in the merger source for the bad timestamp message:

            if (op->Time < LastTimeStamp) {

//     invalid time stamp 

               (*StatsMem[TSSEQERR])++;

               (*StatsMem[TSSEQERR+((LinkNum+1)*MAXCOUNTERS)])++;

               sprintf(message_buffer, "bad timestamp  %d %d 0x%08lx 0x%08lx 0x%016llx 0x%016llx 0x%016llx",LinkNum, link_table[LinkNum]->link_state, op->Data, op->Timestamp, INFO4, op->Time, LastTimeStamp);
      
                report_message(MSG_WARNING);   /***************************/

//              LastTimeStamp = op->Time;


So the message is generated when the merger detects a timewarp.

Took the first warning a data block of errors (The first instance of that particular `LastTimeStamp` and c alculated the time difference between the new timestamp and the last timestamp
The time and LastTimestamp were 0x166bccef3f424a50 0x166bccef4f4232e0 respectively
The time difference between them is 268429456 which is 268ms which seems quite a large difference


06:39 AIDA Crashed
      This time all FEEs were responsive but not showing any stats
      When stopping the DAQ all FEEs except aida06 stopped - attachment 10
      Did a reset of the DAQ and all recovered but no stats on aida06 - attachment 11
      Regained DAQ with a powercycle and a complete restart of the AIDA:8115 Merger and TaperServer

      It is worth noting that aida06 is connected to link 5 the data link which had been producing the bad merge messages overnight.
      We have now had it in aida05, aida6 and aida07. Could it be to do with the correlation scaler rate going into these FEES?

      Going through the var/log/messages on aida-3 aida06 rebooted itself at 06:37
      Mar 13 06:37:16 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Mar 13 06:37:18 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Mar 13 06:37:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)

Looking in /var/log/messages on aida06 no evidence of a reason why:
Mar 12 23:30:56 aida06 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: driver assigned major number 253
Mar 12 23:31:13 aida06 kernel: xaida: open:
Mar 12 23:31:14 aida06 kernel: AIDAMEM: aidamem_open:
Mar 12 23:33:06 aida06 kernel: xaida: open:
Mar 13 05:37:30 aida06 syslogd 1.4.2: restart.
Mar 13 05:37:30 aida06 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Mar 13 05:37:30 aida06 kernel: Using Xilinx Virtex440 machine description
Mar 13 05:37:30 aida06 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
Mar 13 05:37:30 aida06 kernel: Zone PFN ranges:

21.12 DAQ continues OK - file R33_617

      ASIC settings 2019Dec19-16.19.51
      DSSSD#1 slow comparator 0xa
      DSSSD#2 slow comparator 0xa
      DSSSD#3 slow comparator 0xd

      BNC PB-5 Pulser 
      Amplitude1.0V
      Attenuation x1
      Frequency 2Hz
      tau_d 1ms
      - polarity
      Delay 250ns, tail pulse

21.15 System wide checks


FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


		 Base 		Current 	Difference
aida05 fault 	 0x7a58 : 	 0x7a5a : 	 2  
aida06 fault 	 0x65bd : 	 0x65bf : 	 2  
aida07 fault 	 0xcdd6 : 	 0xcdda : 	 4  
aida08 fault 	 0x2ab5 : 	 0x2ab7 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR


			 Base 		Current 		Difference
aida05 fault 	 0x0 : 	 0x1 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

	
Returned 0 0 0 0 0 0 0 0 0 0 0 0  
Mem(KB) :	4	8	16	32	64	128	256	512	1k	2k	4k
aida01 :      5      4      2      0      1      2      3      3      2      3     11   : 55956
aida02 :     21      4      1      2      2      4      2      2      1      4      7   : 40260
aida03 :     10      4      4      1      6      4      1      3      3      3      6   : 36648
aida04 :      7      2      4      0      2      3      2      4      1      4     15   : 73836
aida05 :      1      3      1      5      4      3      1      2      3      4      6   : 37964
aida06 :     12      3      5      2      1      3      3      3      2      4      7   : 41880
aida07 :      2      4      3      3      4      2      2      4      3      3      6   : 37048
aida08 :     17     13      2      3      2      3      2      3      2      3      7   : 39724
aida09 :      3      7      2      0      1      2      2      3      2      2      7   : 37284
aida10 :      2      5      3      2      2      3      1      3      2      3      7   : 39328
aida11 :     23     18      2      3      2      3      3      2      3      3      6   : 36460
aida12 :     18      7      3      1      1      4      2      2      2      3      7   : 39184

      FEE64 Temperatures OK - attachment 1
      Good event statistics OK - attachment 2
      Detector bias & leakage currents OK - attachment 3
      Merger OK - 4.9M data items/s
      TapeServer OK - 16Mb/s

      All histograms zero'd

21.50 At some point between R33_610 (20.59) and R33_625 (21.30) aida05 stopped producing data (zero good events - see attachment 2)

      able to telnet to aida05 - no warnings/error messages in /var/log/messages

      DAQ STOP (all except aida05 stopped OK, aida05 remained GOING)

      issued aida05 reboot command via telnet command line

      DAQ RESET/SETUP/GO (all FEE64s GOING OK except aida05 - zero good events)

21.15 All system wide checks OK *except*

FEE64 module aida09 global clocks failed, 6
 Clock status test result: Passed 11, Failed 1

Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1' 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable


FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module


21.50 aida04 & aida05 statistics for comparison - see attachments 4 & 5

11.00(Germany) System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment1)
      Spectra rates (attachment2)
      FEE temps (attachment3)
      Leakage currents, written to google sheet (attachment4)  
      Merger~ 4.9M items/s
      Tapeserver ~17MB/s 

      In MBS control terminal, connection has been closed intentionally since this morning (file S452f160),
      AIDA has been taken out of the timesorter due to the high data rate, buffers were full
      AIDA cannot be seen in ucesb or Go4.

13.00 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment5)
      Spectra rates (attachment6)
      FEE temps (attachment7)
      Leakage currents, written to google sheet (attachment8)  
      Merger~ 5.1M items/s
      Tapeserver ~18MB/s 

      No timestamp related errors this shift

13.37 AIDA MBS control restarted R33_388
      System wide checks same as previous time

13.54 beam stopped for access,  file R33_396

13.57 seen recent batch of bad timestamp errors in new merger terminal (attachment9) should be around R33_397
      Analysed R33_395, 396, 397, 398 no timewarps
      
14.29 beam back R33_415

14.54 see more bad timestamps in new merger terminal (attachment10)

14.55 System wide checks okay except:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

      Statistics (attachment11)
      Spectra rates (attachment12)
      FEE temps (attachment13)
      Leakage currents, written to google sheet (attachment14)  
      Merger~ 5.0M items/s
      Tapeserver ~17MB/s 

16.05 AIDA included back into timesorter, AIDA scalers now seen in ucesb, file R33_463

16.15 CA takes over until 18:00

17:11 System wide checks:

      FEE64 module aida09 global clocks failed, 6
      Clock status test result: Passed 11, Failed 1
      Understand status as follows
      Status bit 3 : firmware PLL that creates clocks from external clock not locked
      Status bit 2 : always logic '1' 
      Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
      Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
      If all these bits are not set then the operation of the firmware is unreliable

      FEE64 module aida07 failed
      FEE64 module aida09 failed
      Calibration test result: Passed 10, Failed 2
      If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun 
      calibration for that module

		 Base 		Current 	Difference
aida07 fault 	 0xcdd6 : 	 0xcdd7 : 	 1  
White Rabbit error counter test result: Passed 11, Failed 1

17:13 FEE64 temps ok - attachment 15
      Statistics ok - attachment 16
      bias and leakage currents ok - attachment 17