18:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
00:12 Noticed this log in UCESB.
Closed connection [140.181.60.97]...
1 clients...
AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848)
AIDA timewarp is over, skipped 2728 AIDA event(s)
=> Not emitting timewarped event (before 166a7a92fdac5c02)
Recovered from timewarp but skipped 5 event(s)
at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961
Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either
Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message
01:09 Another timewarp message
AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974)
=> Not emitting timewarped event (before 166a83bc6226e9ea)
Recovered from timewarp but skipped 6 event(s)
AIDA timewarp is over, skipped 323841 AIDA event
Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175
Again no timewarps in analysis of file.
02:00 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 4
FEE Temperatures - attachment 5
Bias and leakage currents ok - attachment 6
03:10 Problem with accelerator will be no beam for ~30 minutes
03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place.
There are now also errors in all FEEs for WR events
Base Current Difference
aida01 fault 0xc45f : 0xc46a : 11
aida02 fault 0xd250 : 0xd25b : 11
aida03 fault 0x27b2 : 0x27bd : 11
aida04 fault 0x714b : 0x7156 : 11
aida05 fault 0x74f5 : 0x7500 : 11
aida06 fault 0x558a : 0x5595 : 11
aida07 fault 0x8d20 : 0x8d2b : 11
aida08 fault 0x8131 : 0x813c : 11
aida09 fault 0xf412 : 0xf41d : 11
aida10 fault 0x82c2 : 0x82cd : 11
aida11 fault 0x2845 : 0x2850 : 11
aida12 fault 0x1bf6 : 0x1c01 : 11
White Rabbit error counter test result: Passed 0, Failed 12
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Also errors in FPGA
Base Current Difference
aida03 fault 0x0 : 0x1 : 1
aida04 fault 0x0 : 0x1 : 1
aida10 fault 0x0 : 0x2 : 2
aida11 fault 0x0 : 0x1 : 1
aida12 fault 0x1 : 0x2 : 1
FPGA Timestamp error counter test result: Passed 7, Failed 5
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A8D22 3A5FDE66 , WR/10=> 23DDAE9D2A32FD7, Readout Time => 23DDAE9D3C9C000
aida02 : White Rabbit=> 166A8D22 4C6C3AEB , WR/10=> 23DDAE9D4713917, Readout Time => 23DDAE9D5B94000
aida03 : White Rabbit=> 166A8D22 5FB33A3C , WR/10=> 23DDAE9D65EB906, Readout Time => 23DDAE9D769C000
aida04 : White Rabbit=> 166A8D22 6FBA29DB , WR/10=> 23DDAE9D7F9042F, Readout Time => 23DDAE9D8D14000
aida05 : White Rabbit=> 166A8D22 7F71248D , WR/10=> 23DDAE9D98B5074, Readout Time => 23DDAE9DA9D8000
aida06 : White Rabbit=> 166A8D22 8F78B071 , WR/10=> 23DDAE9DB25AB3E, Readout Time => 23DDAE9DC280000
aida07 : White Rabbit=> 166A8D22 9F5A64B7 , WR/10=> 23DDAE9DCBC3D45, Readout Time => 23DDAE9DDFF0000
aida08 : White Rabbit=> 166A8D22 B1E91092 , WR/10=> 23DDAE9DE974E75, Readout Time => 23DDAE9DFA08000
aida09 : White Rabbit=> 166A8D22 C150D9ED , WR/10=> 23DDAE9E021AF64, Readout Time => 23DDAE9E0FD8000
aida10 : White Rabbit=> 166A8D22 D02BA583 , WR/10=> 23DDAE9E19DF6F3, Readout Time => 23DDAE9E2D50000
aida11 : White Rabbit=> 166A8D22 E287CFC2 , WR/10=> 23DDAE9E373FB2D, Readout Time => 23DDAE9E4720000
aida12 : White Rabbit=> 166A8D22 F2D4C911 , WR/10=> 23DDAE9E515474E, Readout Time => 23DDAE9E1BCC000
System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results
AIDA DAQ Stopped (Still no beam)
The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA).
03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully.
All system wide checks in AIDA now pass. (AIDA06 no longer shows the error)
Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again.
05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7
The problem was with the accelerator operators system. There was no beam.
06:20 Error in WR system wide checks. Rest all ok
This time there is no bad merge items recorded in the merger terminal.
There are also no timewarp events seen in ucesb which were observed with the issues at 3am.
Base Current Difference
aida01 fault 0xdbe9 : 0xdbea : 1
aida02 fault 0xa93d : 0xa93e : 1
aida03 fault 0x8fc4 : 0x8fc5 : 1
aida04 fault 0x3e56 : 0x3e57 : 1
aida05 fault 0xb04c : 0xb04d : 1
aida06 fault 0x5576 : 0x5577 : 1
aida07 fault 0x99ff : 0x9a00 : 1
aida08 fault 0x8412 : 0x8413 : 1
aida09 fault 0x2457 : 0x2458 : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A9523 11478D1D , WR/10=> 23DDBB6B4ED8E1C, Readout Time => 23DDBB6B6148000
aida02 : White Rabbit=> 166A9523 236AA5E8 , WR/10=> 23DDBB6B6BDDD64, Readout Time => 23DDBB6B8164000
aida03 : White Rabbit=> 166A9523 375F235D , WR/10=> 23DDBB6B8BCB6BC, Readout Time => 23DDBB6B9C3C000
aida04 : White Rabbit=> 166A9523 47B8038C , WR/10=> 23DDBB6BA5F338E, Readout Time => 23DDBB6BA77C000
aida05 : White Rabbit=> 166A9523 599FC116 , WR/10=> 23DDBB6BC29934F, Readout Time => 23DDBB6BD34C000
aida06 : White Rabbit=> 166A9523 696EB0A7 , WR/10=> 23DDBB6BDBE44DD, Readout Time => 23DDBB6BECB8000
aida07 : White Rabbit=> 166A9523 7982236B , WR/10=> 23DDBB6BF59D057, Readout Time => 23DDBB6C0418000
aida08 : White Rabbit=> 166A9523 87C85008 , WR/10=> 23DDBB6C0C73B34, Readout Time => 23DDBB6C1C7C000
aida09 : White Rabbit=> 166A9523 97020497 , WR/10=> 23DDBB6C24D0075, Readout Time => 23DDBB6C34B4000
aida10 : White Rabbit=> 166A9523 A69B171D , WR/10=> 23DDBB6C3DC4F1C, Readout Time => 23DDBB6C50A0000
aida11 : White Rabbit=> 166A9523 B86C8D7D , WR/10=> 23DDBB6C5A4748C, Readout Time => 23DDBB6C6C88000
aida12 : White Rabbit=> 166A9523 CA1DD1B2 , WR/10=> 23DDBB6C76961C5, Readout Time => 23DDBB6C5C40000
Possibly occured at 05:28 as a ucesb reports a timewarp at the time.
AIDA Timewarp (166a922b75368928 before 166a922b799e6c92)
AIDA timewarp is over, skipped 2728 AIDA event(s)
Cannot check file as was not writing to file.
Statistics - attachment 8
FEE Temperatures - attachment 9
Bias and leakage currents ok - attachment 10
06:40 There are occasional seconds of beam. According to operators it comes and then it goes again
06:45 Beam fairly stable so we are running again.
AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19
07:00 Beam stopped again
07:02 Beam returned
There are apparently issues with MUSIC chamber one and its resolution
07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing. |