From the installation of the RG58 cable for the 50MHz TTL clock from BRIKEN DAQ
to AIDA Master MACB external clock input we observe DAQ stalls at c. 06.30 and
18.30 daily. This coincides with the nominal start times of the ORNL Clover detector
LN2 fill cycles.

run          start stop  files   recovery

AIDA Master MACB internal clock

R1   28.9.16 15.02 11.20 R1_304  OK

RG174 cable from BRIKEN DAQ 50MHz clock (TTL) to AIDA Master MACB external clock input

     9.10.16 16.55 17.30         no data transfer, TS_SYNC phase = 0, OK
     9.10.16 17.40 19.15 est     no data transfer, TS_SYNC phase = 0, ReSYNC
     9.10.16 22.20 01.40         no data transfer, TS_SYNC phase = 1, ?

    10.10.16 14.20 16.50         no data transfer, TS_SYNC phase = 2, ?
    10.10.16 18.35 18.40/19.40   no data transfer, TS_SYNC phase = 3, some FEE64s stopped later than others, ReSYNC
    10.10.16 20.36 06.35         no data transfer, TS_SYNC phase = 0, power cycle


RG174 cable from BRIKEN DAQ 50MHz clock (TTL) to AIDA Master MACB external clock input
replaced by RG58 cable

run          start stop  files   recovery

R17 11.10.16 15.46 08.22 R17_440 soft reset

R21 12.10.16 14.05 15.30         power dip - power cycle
R22 12.10.16 16.20 19.05         OK 

R24 13.10.16                     OK
R25 13.10.16                     OK
R26 13.10.16                     OK
R27 13.10.16                     OK
R28 13.10.16                     OK
R29 13.10.16                     OK

R32 13.10.16 21.20 22.34 R32_26  soft reset
R33 14.10.16 00.01 00.25 ?       power cycle

R40 14.10.16 01.49 06.26 R40_91  soft reset 
R41 14.10.16                     OK
R42 15.10.16 22.45 07.18 R42_164 System-wide tests - SYNC errors fail *but* SYNC pulses OK - ReSYNC only 
R43 15.10.16 11.39 12.43         disk housekeeping caused merger/tapeserver problem

R46 15.10.16 12.43 18.26 R46_109 soft reset
R47 16.10.16 11.12 18.26 R47_120 power cycle

R49 16.10.16 21.05 06.26 R49_226 soft reset

R51 17.10.16 12.21 18.30 R51_145 soft reset
R52 17.10.16 20.41 06.30 R52_211 soft reset
R53 18.10.16 10.02 18.30 R53_193 soft reset

R55 18.10.16 18.52 19.17 R55_10  soft reset

R60 19.10.16 09.35 16.28 R60_169 OK
R61 19.10.16 16.30 18.26 R61_69  power cycle

R63 19.10.16 20.05 06.46 R63_291 soft reset
R64 20.10.16 09.45 18.25 R64_240 OK - System-wide tests - SYNC errors fail & SYNC pulses fail - other checks OK
R65 20.10.16 18.49 19.17 R65_12  power cycle            
R66 20.10.16 20.02 10.55 R66_396 soft reset  auto LN2 fill may have failed, manually selected c. 11.00
R67 21.10.16 14.02 18.26 R67_121 soft reset
R68 21.10.16 18.40 18.44 R68_1   soft reset

R70 21.10.16 18.53 06.46 R70_315 soft reset
R71 22.10.16 10.42 18.25         OK

R73 22.10.16 19.07 19.17 R73_4   soft reset
R74 22.10.16 19.38 06.31 R74_296 soft reset

R76 23.10.16 10.46 11.52 R76_29  merger crashed, soft reset
R77 23.10.16 18.50 19.17         disk housekeeping caused merger/tapeserver problem
R78 23.10.16 19.47 06.46 R78_291 soft reset
R79 24.10.16 11.07 18.46 R79_182 soft reset
R80 24.10.16 18.56 19.17 R80_5   DAQ stall, soft reset
R81 24.10.16 19.29 06.26 R81_169 soft reset
R82 25.10.16 10.22 18.30 R82_125 -> safe state

R83 27.10.16 14.35 18.30 R83_125 soft reset
R86 27.10.16 20.48 06.26 R86_245 soft reset
R87 28.10.16 14.34 18.47 R87_274 soft reset

R89 29.10.16 19.10 07.17 R89_102 power cycle

soft reset => no FEE64 power cycle
      Restart merger
      DAQ STOP
      System wide checks 
      DAQ RESET & SETUP
      System-wide checks
      Disable H & data transfer #1
      Restart merger & tapeserver
      Enable H & data transfer #1