AIDA
GELINA
BRIKEN
nToF
CRIB
ISOLDE
CIRCE
nTOFCapture
DESPEC
DTAS
EDI_PSA
179Ta
CARME
StellarModelling
DCF
K40
DESPEC
Draft saved at 00:00:00
Fields marked with
*
are required
Entry time:
Mon Mar 8 22:51:47 2021
Author
*
:
Subject
*
:
18:00 ASIC settings 2019Dec19-16.19.51 DSSSD#1 slow comparator 0xa DSSSD#2 slow comparator 0xa DSSSD#3 slow comparator 0xd BNC PB-5 Pulser Amplitude1.0V Attenuation x1 Frequency 2Hz tau_d 1ms - polarity Delay 250ns, tail pulse All system wide checks ok *except aida06 fails clock status 6* Statistics - attachment 1 FEE Temperatures - attachment 2 Bias and leakage currents ok - attachment 3 00:12 Noticed this log in UCESB. Closed connection [140.181.60.97]... 1 clients... AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848) AIDA timewarp is over, skipped 2728 AIDA event(s) => Not emitting timewarped event (before 166a7a92fdac5c02) Recovered from timewarp but skipped 5 event(s) at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961 Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message 01:09 Another timewarp message AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974) => Not emitting timewarped event (before 166a83bc6226e9ea) Recovered from timewarp but skipped 6 event(s) AIDA timewarp is over, skipped 323841 AIDA event Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175 Again no timewarps in analysis of file. 02:00 All system wide checks ok *except aida06 fails clock status 6* Statistics - attachment 4 FEE Temperatures - attachment 5 Bias and leakage currents ok - attachment 6 03:10 Problem with accelerator will be no beam for ~30 minutes 03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place. There are now also errors in all FEEs for WR events Base Current Difference aida01 fault 0xc45f : 0xc46a : 11 aida02 fault 0xd250 : 0xd25b : 11 aida03 fault 0x27b2 : 0x27bd : 11 aida04 fault 0x714b : 0x7156 : 11 aida05 fault 0x74f5 : 0x7500 : 11 aida06 fault 0x558a : 0x5595 : 11 aida07 fault 0x8d20 : 0x8d2b : 11 aida08 fault 0x8131 : 0x813c : 11 aida09 fault 0xf412 : 0xf41d : 11 aida10 fault 0x82c2 : 0x82cd : 11 aida11 fault 0x2845 : 0x2850 : 11 aida12 fault 0x1bf6 : 0x1c01 : 11 White Rabbit error counter test result: Passed 0, Failed 12 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR Also errors in FPGA Base Current Difference aida03 fault 0x0 : 0x1 : 1 aida04 fault 0x0 : 0x1 : 1 aida10 fault 0x0 : 0x2 : 2 aida11 fault 0x0 : 0x1 : 1 aida12 fault 0x1 : 0x2 : 1 FPGA Timestamp error counter test result: Passed 7, Failed 5 If any of these counts are reported as in error The ASIC readout system has detected a timeslip. That is the timestamp read from the time FIFO is not younger than the last The following are timestamp values from each of the FEEs taken in sequence If time does not increase in a reasonable manner run the system wide checks aida01 : White Rabbit=> 166A8D22 3A5FDE66 , WR/10=> 23DDAE9D2A32FD7, Readout Time => 23DDAE9D3C9C000 aida02 : White Rabbit=> 166A8D22 4C6C3AEB , WR/10=> 23DDAE9D4713917, Readout Time => 23DDAE9D5B94000 aida03 : White Rabbit=> 166A8D22 5FB33A3C , WR/10=> 23DDAE9D65EB906, Readout Time => 23DDAE9D769C000 aida04 : White Rabbit=> 166A8D22 6FBA29DB , WR/10=> 23DDAE9D7F9042F, Readout Time => 23DDAE9D8D14000 aida05 : White Rabbit=> 166A8D22 7F71248D , WR/10=> 23DDAE9D98B5074, Readout Time => 23DDAE9DA9D8000 aida06 : White Rabbit=> 166A8D22 8F78B071 , WR/10=> 23DDAE9DB25AB3E, Readout Time => 23DDAE9DC280000 aida07 : White Rabbit=> 166A8D22 9F5A64B7 , WR/10=> 23DDAE9DCBC3D45, Readout Time => 23DDAE9DDFF0000 aida08 : White Rabbit=> 166A8D22 B1E91092 , WR/10=> 23DDAE9DE974E75, Readout Time => 23DDAE9DFA08000 aida09 : White Rabbit=> 166A8D22 C150D9ED , WR/10=> 23DDAE9E021AF64, Readout Time => 23DDAE9E0FD8000 aida10 : White Rabbit=> 166A8D22 D02BA583 , WR/10=> 23DDAE9E19DF6F3, Readout Time => 23DDAE9E2D50000 aida11 : White Rabbit=> 166A8D22 E287CFC2 , WR/10=> 23DDAE9E373FB2D, Readout Time => 23DDAE9E4720000 aida12 : White Rabbit=> 166A8D22 F2D4C911 , WR/10=> 23DDAE9E515474E, Readout Time => 23DDAE9E1BCC000 System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results AIDA DAQ Stopped (Still no beam) The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA). 03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully. All system wide checks in AIDA now pass. (AIDA06 no longer shows the error) Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again. 05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7 The problem was with the accelerator operators system. There was no beam. 06:20 Error in WR system wide checks. Rest all ok This time there is no bad merge items recorded in the merger terminal. There are also no timewarp events seen in ucesb which were observed with the issues at 3am. Base Current Difference aida01 fault 0xdbe9 : 0xdbea : 1 aida02 fault 0xa93d : 0xa93e : 1 aida03 fault 0x8fc4 : 0x8fc5 : 1 aida04 fault 0x3e56 : 0x3e57 : 1 aida05 fault 0xb04c : 0xb04d : 1 aida06 fault 0x5576 : 0x5577 : 1 aida07 fault 0x99ff : 0x9a00 : 1 aida08 fault 0x8412 : 0x8413 : 1 aida09 fault 0x2457 : 0x2458 : 1 White Rabbit error counter test result: Passed 3, Failed 9 Understand the status reports as follows:- Status bit 3 : White Rabbit decoder detected an error in the received data Status bit 2 : Firmware registered WR error, no reload of Timestamp Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR The following are timestamp values from each of the FEEs taken in sequence If time does not increase in a reasonable manner run the system wide checks aida01 : White Rabbit=> 166A9523 11478D1D , WR/10=> 23DDBB6B4ED8E1C, Readout Time => 23DDBB6B6148000 aida02 : White Rabbit=> 166A9523 236AA5E8 , WR/10=> 23DDBB6B6BDDD64, Readout Time => 23DDBB6B8164000 aida03 : White Rabbit=> 166A9523 375F235D , WR/10=> 23DDBB6B8BCB6BC, Readout Time => 23DDBB6B9C3C000 aida04 : White Rabbit=> 166A9523 47B8038C , WR/10=> 23DDBB6BA5F338E, Readout Time => 23DDBB6BA77C000 aida05 : White Rabbit=> 166A9523 599FC116 , WR/10=> 23DDBB6BC29934F, Readout Time => 23DDBB6BD34C000 aida06 : White Rabbit=> 166A9523 696EB0A7 , WR/10=> 23DDBB6BDBE44DD, Readout Time => 23DDBB6BECB8000 aida07 : White Rabbit=> 166A9523 7982236B , WR/10=> 23DDBB6BF59D057, Readout Time => 23DDBB6C0418000 aida08 : White Rabbit=> 166A9523 87C85008 , WR/10=> 23DDBB6C0C73B34, Readout Time => 23DDBB6C1C7C000 aida09 : White Rabbit=> 166A9523 97020497 , WR/10=> 23DDBB6C24D0075, Readout Time => 23DDBB6C34B4000 aida10 : White Rabbit=> 166A9523 A69B171D , WR/10=> 23DDBB6C3DC4F1C, Readout Time => 23DDBB6C50A0000 aida11 : White Rabbit=> 166A9523 B86C8D7D , WR/10=> 23DDBB6C5A4748C, Readout Time => 23DDBB6C6C88000 aida12 : White Rabbit=> 166A9523 CA1DD1B2 , WR/10=> 23DDBB6C76961C5, Readout Time => 23DDBB6C5C40000 Possibly occured at 05:28 as a ucesb reports a timewarp at the time. AIDA Timewarp (166a922b75368928 before 166a922b799e6c92) AIDA timewarp is over, skipped 2728 AIDA event(s) Cannot check file as was not writing to file. Statistics - attachment 8 FEE Temperatures - attachment 9 Bias and leakage currents ok - attachment 10 06:40 There are occasional seconds of beam. According to operators it comes and then it goes again 06:45 Beam fairly stable so we are running again. AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19 07:00 Beam stopped again 07:02 Beam returned There are apparently issues with MUSIC chamber one and its resolution 07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing.
Encoding
:
HTML
ELCode
plain
Suppress Email notification
Resubmit as new entry
Attachment 1:
210308_2354_Stats.png
Original size: 1910x402
Attachment 2:
210308_2355_Temp.png
Original size: 1908x473
Attachment 3:
210308_2357_Bias.png
Original size: 496x314
Attachment 4:
210309_0104_Stats.png
Original size: 1909x317
Attachment 5:
210309_0205_Temp.png
Original size: 1911x461
Attachment 6:
210309_0205_Bias.png
Original size: 499x318
Attachment 7:
210309_0529_layout1.png
Original size: 1887x823
Attachment 8:
210309_0619_Stats.png
Original size: 1911x430
Attachment 9:
210309_0620_Temp.png
Original size: 1906x501
Attachment 10:
210309_0623_Bias.png
Original size: 496x317
Attachment 11:
Drop attachments here...
Draft saved at 00:00:00
ELOG V3.1.4-unknown