ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required
Entry time:	Sun Mar 30 09:18:30 2025
Author*:
Subject*:
> 18:00 ASIC settings 2019Dec19-16.19.51 > DSSSD#1 slow comparator 0xa > DSSSD#2 slow comparator 0xa > DSSSD#3 slow comparator 0xd > > BNC PB-5 Pulser > Amplitude1.0V > Attenuation x1 > Frequency 2Hz > tau_d 1ms > - polarity > Delay 250ns, tail pulse > > All system wide checks ok except aida06 fails clock status 6 > Statistics - attachment 1 > FEE Temperatures - attachment 2 > Bias and leakage currents ok - attachment 3 > > 00:12 Noticed this log in UCESB. > Closed connection [140.181.60.97]... > 1 clients... > AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848) > AIDA timewarp is over, skipped 2728 AIDA event(s) > => Not emitting timewarped event (before 166a7a92fdac5c02) > Recovered from timewarp but skipped 5 event(s) > > at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961 > Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either > Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message > > 01:09 Another timewarp message > AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974) > => Not emitting timewarped event (before 166a83bc6226e9ea) > Recovered from timewarp but skipped 6 event(s) > AIDA timewarp is over, skipped 323841 AIDA event > > Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175 > Again no timewarps in analysis of file. > > > 02:00 All system wide checks ok except aida06 fails clock status 6 > Statistics - attachment 4 > FEE Temperatures - attachment 5 > Bias and leakage currents ok - attachment 6 > > 03:10 Problem with accelerator will be no beam for ~30 minutes > > > 03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place. > There are now also errors in all FEEs for WR events > > Base Current Difference > aida01 fault 0xc45f : 0xc46a : 11 > aida02 fault 0xd250 : 0xd25b : 11 > aida03 fault 0x27b2 : 0x27bd : 11 > aida04 fault 0x714b : 0x7156 : 11 > aida05 fault 0x74f5 : 0x7500 : 11 > aida06 fault 0x558a : 0x5595 : 11 > aida07 fault 0x8d20 : 0x8d2b : 11 > aida08 fault 0x8131 : 0x813c : 11 > aida09 fault 0xf412 : 0xf41d : 11 > aida10 fault 0x82c2 : 0x82cd : 11 > aida11 fault 0x2845 : 0x2850 : 11 > aida12 fault 0x1bf6 : 0x1c01 : 11 > White Rabbit error counter test result: Passed 0, Failed 12 > > Understand the status reports as follows:- > Status bit 3 : White Rabbit decoder detected an error in the received data > Status bit 2 : Firmware registered WR error, no reload of Timestamp > Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR > > Also errors in FPGA > Base Current Difference > aida03 fault 0x0 : 0x1 : 1 > aida04 fault 0x0 : 0x1 : 1 > aida10 fault 0x0 : 0x2 : 2 > aida11 fault 0x0 : 0x1 : 1 > aida12 fault 0x1 : 0x2 : 1 > FPGA Timestamp error counter test result: Passed 7, Failed 5 > If any of these counts are reported as in error > The ASIC readout system has detected a timeslip. > That is the timestamp read from the time FIFO is not younger than the last > > The following are timestamp values from each of the FEEs taken in sequence > If time does not increase in a reasonable manner run the system wide checks > > aida01 : White Rabbit=> 166A8D22 3A5FDE66 , WR/10=> 23DDAE9D2A32FD7, Readout Time => 23DDAE9D3C9C000 > aida02 : White Rabbit=> 166A8D22 4C6C3AEB , WR/10=> 23DDAE9D4713917, Readout Time => 23DDAE9D5B94000 > aida03 : White Rabbit=> 166A8D22 5FB33A3C , WR/10=> 23DDAE9D65EB906, Readout Time => 23DDAE9D769C000 > aida04 : White Rabbit=> 166A8D22 6FBA29DB , WR/10=> 23DDAE9D7F9042F, Readout Time => 23DDAE9D8D14000 > aida05 : White Rabbit=> 166A8D22 7F71248D , WR/10=> 23DDAE9D98B5074, Readout Time => 23DDAE9DA9D8000 > aida06 : White Rabbit=> 166A8D22 8F78B071 , WR/10=> 23DDAE9DB25AB3E, Readout Time => 23DDAE9DC280000 > aida07 : White Rabbit=> 166A8D22 9F5A64B7 , WR/10=> 23DDAE9DCBC3D45, Readout Time => 23DDAE9DDFF0000 > aida08 : White Rabbit=> 166A8D22 B1E91092 , WR/10=> 23DDAE9DE974E75, Readout Time => 23DDAE9DFA08000 > aida09 : White Rabbit=> 166A8D22 C150D9ED , WR/10=> 23DDAE9E021AF64, Readout Time => 23DDAE9E0FD8000 > aida10 : White Rabbit=> 166A8D22 D02BA583 , WR/10=> 23DDAE9E19DF6F3, Readout Time => 23DDAE9E2D50000 > aida11 : White Rabbit=> 166A8D22 E287CFC2 , WR/10=> 23DDAE9E373FB2D, Readout Time => 23DDAE9E4720000 > aida12 : White Rabbit=> 166A8D22 F2D4C911 , WR/10=> 23DDAE9E515474E, Readout Time => 23DDAE9E1BCC000 > > System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results > > AIDA DAQ Stopped (Still no beam) > The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA). > > 03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully. > > All system wide checks in AIDA now pass. (AIDA06 no longer shows the error) > Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again. > > 05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7 > The problem was with the accelerator operators system. There was no beam. > > 06:20 Error in WR system wide checks. Rest all ok > This time there is no bad merge items recorded in the merger terminal. > There are also no timewarp events seen in ucesb which were observed with the issues at 3am. > > Base Current Difference > aida01 fault 0xdbe9 : 0xdbea : 1 > aida02 fault 0xa93d : 0xa93e : 1 > aida03 fault 0x8fc4 : 0x8fc5 : 1 > aida04 fault 0x3e56 : 0x3e57 : 1 > aida05 fault 0xb04c : 0xb04d : 1 > aida06 fault 0x5576 : 0x5577 : 1 > aida07 fault 0x99ff : 0x9a00 : 1 > aida08 fault 0x8412 : 0x8413 : 1 > aida09 fault 0x2457 : 0x2458 : 1 > White Rabbit error counter test result: Passed 3, Failed 9 > > Understand the status reports as follows:- > Status bit 3 : White Rabbit decoder detected an error in the received data > Status bit 2 : Firmware registered WR error, no reload of Timestamp > Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR > > > The following are timestamp values from each of the FEEs taken in sequence > If time does not increase in a reasonable manner run the system wide checks > > aida01 : White Rabbit=> 166A9523 11478D1D , WR/10=> 23DDBB6B4ED8E1C, Readout Time => 23DDBB6B6148000 > aida02 : White Rabbit=> 166A9523 236AA5E8 , WR/10=> 23DDBB6B6BDDD64, Readout Time => 23DDBB6B8164000 > aida03 : White Rabbit=> 166A9523 375F235D , WR/10=> 23DDBB6B8BCB6BC, Readout Time => 23DDBB6B9C3C000 > aida04 : White Rabbit=> 166A9523 47B8038C , WR/10=> 23DDBB6BA5F338E, Readout Time => 23DDBB6BA77C000 > aida05 : White Rabbit=> 166A9523 599FC116 , WR/10=> 23DDBB6BC29934F, Readout Time => 23DDBB6BD34C000 > aida06 : White Rabbit=> 166A9523 696EB0A7 , WR/10=> 23DDBB6BDBE44DD, Readout Time => 23DDBB6BECB8000 > aida07 : White Rabbit=> 166A9523 7982236B , WR/10=> 23DDBB6BF59D057, Readout Time => 23DDBB6C0418000 > aida08 : White Rabbit=> 166A9523 87C85008 , WR/10=> 23DDBB6C0C73B34, Readout Time => 23DDBB6C1C7C000 > aida09 : White Rabbit=> 166A9523 97020497 , WR/10=> 23DDBB6C24D0075, Readout Time => 23DDBB6C34B4000 > aida10 : White Rabbit=> 166A9523 A69B171D , WR/10=> 23DDBB6C3DC4F1C, Readout Time => 23DDBB6C50A0000 > aida11 : White Rabbit=> 166A9523 B86C8D7D , WR/10=> 23DDBB6C5A4748C, Readout Time => 23DDBB6C6C88000 > aida12 : White Rabbit=> 166A9523 CA1DD1B2 , WR/10=> 23DDBB6C76961C5, Readout Time => 23DDBB6C5C40000 > > Possibly occured at 05:28 as a ucesb reports a timewarp at the time. > AIDA Timewarp (166a922b75368928 before 166a922b799e6c92) > AIDA timewarp is over, skipped 2728 AIDA event(s) > Cannot check file as was not writing to file. > > Statistics - attachment 8 > FEE Temperatures - attachment 9 > Bias and leakage currents ok - attachment 10 > > 06:40 There are occasional seconds of beam. According to operators it comes and then it goes again > > 06:45 Beam fairly stable so we are running again. > AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19 > > 07:00 Beam stopped again > 07:02 Beam returned > There are apparently issues with MUSIC chamber one and its resolution > > 07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing. Encoding: HTML ELCode plain Suppress Email notification

Attachment 1:
Drop attachments here...

ELOG V3.1.4-unknown