AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40
  DESPEC  ELOG logo
Message ID: 618     Entry time: Sun Apr 28 16:07:59 2024
Author: TD 
Subject: [How To] Recover DAQ when one FEE64 stops sending data - short(er) version 
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5)   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

6) Start DAQ

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

8) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

9) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

10) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.
ELOG V3.1.4-unknown