08:09 DAQ crashes at start of shift (!)

      again happens at same time as beam being lost

      All FEE64 lost connection

      Beam downtime for next 3-4 hours

      Performed powercycle and reset

08:40 AIDA back up and running, writing to R36

08:42 *all* system wide checks ok

      temperatures ok - attachment 1

      statistics ok - attachment 2

      detector bias/leakage currents ok - attachment 3

08:57 options file size check;


 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Tue Apr 20 08:22:14 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

08:59 no error messages in database check terminal
      stats ok

09:16 waveforms - attachments 4 & 5

09:30 no error messages in database check terminal
      stats ok

10:07 no error messages in database check terminal
      stats ok

10:30 aida01, aida02, aida03, aida04, aida07 lost connection, DAQ crashes

      powercycle attempted, but FEE modules remain unconnected

      TD performs further power cycle

11.12 reboot aida05

      DAQ reset/setup

      system wide checks

      sync asic clocks OK
      clock status & fpga timestamp errors OK
      
      WR decoder status

		 Base 		Current 	Difference
aida09 fault 	 0xb307 : 	 0xb307 : 	 0  
aida09 : WR status 0x40
 aida10 fault 	 0x3d6c : 	 0x3d6c : 	 0  
aida10 : WR status 0x40
 aida11 fault 	 0xa610 : 	 0xa610 : 	 0  
aida11 : WR status 0x40
 aida12 fault 	 0x7c7 : 	 0x7c7 : 	 0  
aida12 : WR status 0x40
 White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

NH had access to area, unplugged and replugged HDMI cables from aida09, aida10, aida11, aida12 root to MACB (at both ends)

earlier on, Akash secured HDMI cable to aida07

11:55 AIDA powercycle and reset, DAQ ok now

      writing to file NULL/R45

12:00 CB takes over