ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required

Entry time:

Tue Apr 20 07:06:20 2021

Author*:

Subject*:

08:09 DAQ crashes at start of shift (!)

again happens at same time as beam being lost

All FEE64 lost connection

Beam downtime for next 3-4 hours

Performed powercycle and reset

08:40 AIDA back up and running, writing to R36

08:42 *all* system wide checks ok

temperatures ok - attachment 1

statistics ok - attachment 2

detector bias/leakage currents ok - attachment 3

08:57 options file size check;

Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

FEE : aida01 =>   Options file size is 1025 	 Last changed Tue Apr 20 08:22:14 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Mon Apr 19 09:05:25 CEST 2021

08:59 no error messages in database check terminal
      stats ok

09:16 waveforms - attachments 4 & 5

09:30 no error messages in database check terminal
      stats ok

10:07 no error messages in database check terminal
      stats ok

10:30 aida01, aida02, aida03, aida04, aida07 lost connection, DAQ crashes

powercycle attempted, but FEE modules remain unconnected

TD performs further power cycle

11.12 reboot aida05

DAQ reset/setup

system wide checks

sync asic clocks OK
      clock status & fpga timestamp errors OK
      
      WR decoder status

Base 		Current 	Difference
aida09 fault 	 0xb307 : 	 0xb307 : 	 0  
aida09 : WR status 0x40
 aida10 fault 	 0x3d6c : 	 0x3d6c : 	 0  
aida10 : WR status 0x40
 aida11 fault 	 0xa610 : 	 0xa610 : 	 0  
aida11 : WR status 0x40
 aida12 fault 	 0x7c7 : 	 0x7c7 : 	 0  
aida12 : WR status 0x40
 White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

NH had access to area, unplugged and replugged HDMI cables from aida09, aida10, aida11, aida12 root to MACB (at both ends)

earlier on, Akash secured HDMI cable to aida07

11:55 AIDA powercycle and reset, DAQ ok now

writing to file NULL/R45

12:00 CB takes over

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1: