ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required
Entry time:	Sat Mar 29 11:30:03 2025
Author*:
Subject*:
> 08:16 System wide checks all ok > Statistics - attachment 1 > Temperatures - attachment 2 > bias - attachment 3 > Merger running 4.5e6 events per second > Tape data writing at 42MB per second > > Currently no beam > > 09:45 While there is no beam we will perform a longer test of the new merger. > Run R20 stopped and merger changed to neew version > Started R21. > > The now usual behaviour of additional files at start of merge observed. > N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was > set to going > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_0 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_1 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_2 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_3 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_4 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_5 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_6 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_7 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_8 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_9 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_10 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_11 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_12 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_13 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_14 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_15 > -rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_16 > -rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17 > -rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18 > > TapeData rate of 13280 kB/sec > No errors observed in ucesb > Correlations seen between time machine in AIDA and Ge in online > Pulser rate in the online makes sense > With the current data rate remaining HDD space on current drive will last around 89 hours. > > 09.20 analysis of file R21_27 > ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data > > > 10:24 Message while performing system wide checks: > Get returned with an error > error: SOAP http transport timed out after 10000 ms > NONE > error: SOAP http transport timed out after 10000 ms > while executing > "$transport $procVarName $url $req" > (procedure "::SOAP::invoke" line 18) > invoked from within > "::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10" > ("eval" body line 1) > invoked from within > "eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args" > (procedure "XAIDAAccessClient__Get" line 1) > invoked from within > "XAIDAAccessClient__Get $Addr" > > aida02 restarted itself during the reset process > Looking at the log messages on aida02 cannot see any reason for the cause of the restart. > System wide checks following the restart all ok. > When restarting the MBS relay errors observed until a timestamp was observed: > Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000 > > 10:55 Statistics - attachment 5 > Temperature - attachment 6 > Bias - attachment 7 > > 11:03 Time machine correlation spectra with new merger > AIDA - FATIMA - Attachment 8 > AIDA - Ge - Attachment 9 > > 11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the > layout to determine which FEE/ASIC caused the events before checking ASIC control > > 12:12 System wide checks all ok except adc calibration which is same as before > Statistics - attachment 10 > Temperature - attachment 11 > Bias and leakage currents ok - attachment 12 > > A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat. > > After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time. > They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon > cable from the VME scalers. > Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of > DSSD3. > > 12:50 We have waveforms for some FEES > Layout 7 attachment -14 > Layout 8 - attachment 15 > > 14.00 (LS) > Still no beam > System wide checks okay except same as before: > FEE64 module aida07 failed > FEE64 module aida10 failed > Calibration test result: Passed 10, Failed 2 > If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun > calibration for that module* > > Statistics (attachment16) > Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16): > aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up > aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on > > FEE Temps (attachment17) > Leakage currents written to sheets(attachment18) > Merger ~ 45M items/s > TapeServer~ 14MB/s > > 14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file > R22_98, rate spectra attached (attachment19) > > 14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw > no timewarps so we are not losing anything. > Restarted MBS relay, errors have not reappeared so far > > 16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks) > Statistics (attachment20) > Rate spectra (attachment21) > FEE Temps (attachment22) > Leakage currents written to sheets(attachment23) > Merger ~ 4.3M items/s > TapeServer~ 14MB/s > Current file R22_148 > > 16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from > S452f115 > Around R22_150 > > 18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm) > Rates of R22_122 (attachment25): > First DSSSD (fee0-fee3) ~66/s > Second DSSSD (fee4-fee7) ~49/s > Third DSSSD (fee8-fee11) ~22/s > > Rates of R22_178 (attachment26): > First DSSSD (fee0-fee3) ~122/s > Second DSSSD (fee4-fee7) ~94/s > Third DSSSD (fee8-fee11) ~49/s > > > 18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks) > Statistics (attachment27) > Rate spectra (attachment28) > FEE Temps (attachment29) > Leakage currents written to sheets(attachment30), look to be on the way down > Merger ~ 4.5M items/s > TapeServer~ 15MB/s > > 18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier > > 19:34 MBS DAQ Crashed > 20:00 DAQ is still down. They have Sultan working on it. > > 20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre > reboot) > We are borrowing a HDD from the SHIP group > > 20:30 System wide checks all ok > Statistics - attachment 31 > Temperature - attachment 32 > Bias - attachment 33 > > 22:10 MBS DAQ has crashed > Merger terminal is now showing a large number of bad merge events > > It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS. > > > 22:42 System wide checks all ok > Statistics - attachment 34 > Temperature - Attachment 35 > Bias and leakage currents ok - attachment 36 > > 23:33 System wide checks all ok - attachment 37 > Temperatures - attachment 38 > Bias and leakage currents ok - attachment 39 Encoding: HTML ELCode plain Suppress Email notification

Attachment 1:
Drop attachments here...

ELOG V3.1.4-unknown