ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required
Entry time:	Sat Mar 29 11:24:29 2025
Author*:
Subject*:
> > Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running and data is being transferred to the Merger and TapeServer. > > Typical symptoms are: > > - no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser'] > - zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics'] > - Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control'] > active data links are green, inactive links are red > N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n > > 0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ > > 1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control'] > all FEE64s should change status from 'going' to 'stopped' except whichever FEE64(s) have a problem - typically their status is 'going' or 'error' > > 2) Restart Merger by selecting the Merger icon [workspace 3, display 2] > terminal window showing the startup will popup > > 3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control'] > > 4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control'] > all data links will be inactive (red) > > 5) Reboot problem FEE64s > Launch a command terminal and telnet to the problem FEE64 for example > > telnet aida04 > > Enter user 'root' and enter password. When logged in reboot the FEE64 using the command > > reboot > > 6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal'] > you should see two messages showing the FEE64 mounting filesystems > wait c. three minutes for boot to complete > > 7) Restart DAQ > > Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control'] > Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control'] > Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control'] > Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control'] > Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control'] > Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control'] > > 8) Restore ASIC settings > > Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control'] > Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings'] > Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings'] > Wait 10s > > 9) Check ASIC settings > > Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control'] > Wait for popup indicating completion of checks and dismiss > > 10) System wide checks > > Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks'] > Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks'] > Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks'] > > Select 'Check clock status' [workspace 2, display 1, browser tab 'System Wide Checks'] > Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks'] > Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks'] > make a note or take a screenshot of any error messages observed > > 11) Start DAQ > > Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control'] > Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control'] > > Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control'] > check that DAQ is actually going - if not repeat 'GO' > > 12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control'] > readout times will be displayed for each FEE64 > check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support > > 13) Stop DAQ > > Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control'] > check that DAQ has actually stopped - if not repeat 'STOP > > 14) Enable data transfer to Merger and TapeServer and start DAQ > > Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control'] > > Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control'] > check that DAQ is actually going - if not repeat 'GO' > > 15) Check data is being transferred to Merger and TapeServer > > Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control'] > you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads) > > Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer'] > you should see 'Blocks written' and 'kb written' counters increase > > 16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal] > > 17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data. > > If not, please request expert assistance. > > > > > > > > > > > > > > > > > > > > > > > > > Encoding: HTML ELCode plain Suppress Email notification

Attachment 1:
Drop attachments here...

ELOG V3.1.4-unknown