ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required
Entry time:	Fri Apr 26 08:15:59 2024
Author*:
Subject*:
Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running and data is being transferred to the Merger and TapeServer. Typical symptoms are: - no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser'] - zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics'] - Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control'] active data links are green, inactive links are red N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n 0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ 1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control'] all FEE64s should change status from 'going' to 'stopped' except whichever FEE64(s) have a problem - typically their status is 'going' or 'error' 2) Restart Merger by selecting the Merger icon [workspace 3, display 2] terminal window showing the startup will popup 3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control'] 4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control'] all data links will be inactive (red) 5) Reboot problem FEE64s Launch a command terminal and telnet to the problem FEE64 for example telnet aida04 Enter user 'root' and enter password. When logged in reboot the FEE64 using the command reboot 6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal'] you should see two messages showing the FEE64 mounting filesystems wait c. three minutes for boot to complete 7) Restart DAQ Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control'] Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control'] Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control'] Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control'] Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control'] Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control'] 8) Restore ASIC settings Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control'] Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings'] Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings'] Wait 10s 9) Check ASIC settings Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control'] Wait for popup indicating completion of checks and dismiss 10) System wide checks Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks'] Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks'] Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks'] Select 'Check clock status' [workspace 2, display 1, browser tab 'System Wide Checks'] Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks'] Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks'] make a note or take a screenshot of any error messages observed 11) Start DAQ Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control'] Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control'] Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control'] check that DAQ is actually going - if not repeat 'GO' 12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control'] readout times will be displayed for each FEE64 check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support 13) Stop DAQ Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control'] check that DAQ has actually stopped - if not repeat 'STOP 14) Enable data transfer to Merger and TapeServer and start DAQ Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control'] Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control'] check that DAQ is actually going - if not repeat 'GO' 15) Check data is being transferred to Merger and TapeServer Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control'] you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads) Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer'] you should see 'Blocks written' and 'kb written' counters increase 16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal] 17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data. If not, please request expert assistance. Encoding: HTML ELCode plain Suppress Email notification Resubmit as new entry

Attachment 1:
Drop attachments here...

ELOG V3.1.4-unknown