Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running *and* data is being transferred to the Merger and TapeServer.

Typical symptoms are:

- no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser']
- zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics']
- Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control']
   active data links are green, inactive links are red
   N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n

0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ

1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
   all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'

2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
    terminal window showing the startup will popup

3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']

4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
     all data links will be inactive (red)

5) Reboot problem FEE64s
   Launch a command terminal and telnet to the problem FEE64 for example

    telnet aida04

   Enter user 'root' and enter password. When logged in reboot the FEE64 using the command

    reboot

6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal']
    you should see two messages showing the FEE64 mounting filesystems
    wait c. three minutes for boot to complete

7) Restart DAQ

   Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control']
   Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control']
   Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control']
   Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control']
   Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control']
   Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']

8) Restore ASIC settings

   Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control']
   Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings']
   Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings']
   Wait 10s

9) Check ASIC settings

   Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control']
   Wait for popup indicating completion of checks and dismiss

10) System wide checks

    Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']

    Select 'Check clock status'  [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks']
    Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks']
     make a note or take a screenshot of any error messages observed

11) Start DAQ

    Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control']
    Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
     readout times will be displayed for each FEE64
     check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support

13) Stop DAQ 

    Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ has *actually* stopped - if not repeat 'STOP

14) Enable data transfer to Merger and TapeServer and start DAQ

    Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']

    Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
     check that DAQ is *actually* going - if not repeat 'GO'

15) Check data is being transferred to Merger and TapeServer

    Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
     you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)

    Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
     you should see 'Blocks written' and 'kb written' counters increase

16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]

17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.

    If not, please request expert assistance.