00:05 DESPEC tuning beam, changing spill length

00:21 DAQ continues ok
      Merger/TapeServer ok

00:25 still no beam in AIDA

00:30 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 1

00:52 beam in AIDA - rates spectra - attachment 2

01:05 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 3

      beam is off again

01:30 beam back

01:32 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok - attachment 4

01:37 beam off - FRS increasing spill length to 5s

01:48 beam back with new spill setting - 5s on spill, 2s off spill

02:07 system wide checks ok except;

WR decoder status

      		 Base 		Current 	Difference
aida05 fault 	 0xc879 : 	 0xc87b : 	 2  
aida06 fault 	 0x323c : 	 0x323e : 	 2  
aida07 fault 	 0xfb3a : 	 0xfb42 : 	 8  
aida08 fault 	 0xd3d6 : 	 0xd3d8 : 	 2  
White Rabbit error counter test result: Passed 8, Failed 4

Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

FPGA check :

			 Base 		Current 		Difference
aida07 fault 	 0xf : 	 0x10 : 	 1  
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last

02:10 FEE64 Temps ok - attachment 5

02:20 good event statistics ok - attachment 6

02:27 detector bias / leakage currents ok - attachment 7

02:30 statistics ok, all FEE64 showing rates

      no error messages in database check

      AIDA scalars on ucesb ok

      beam off

03:00 stats/ucesb/database checks ok

03:30 stats/ucesb/database checks ok

      still no beam in AIDA

05:11 system is checked, beam is back with low intensity 

05:13 FEE64 Temps ok - attachment 8

05:13 good event statistics ok - attachment 9

05:13 detector bias / leakage currents ok - attachment 10

05:33 stats/ucesb/database checks ok

06:00 AIDA crashes, lose connection to aida09

06:20 aida08 does not respond on powercycle and reset

06:30 Called TD 

06:50 successfully reset AIDA. Writing to file R30

      synchronise ASIC clocks:
	
FEE64 module aida01  => 	7
FEE64 module aida02  => 	7
FEE64 module aida03  => 	7
FEE64 module aida04  => 	7
FEE64 module aida05  => 	7
FEE64 module aida06  => 	7
FEE64 module aida07  => 	7
FEE64 module aida08  => 	7
FEE64 module aida09  => 	7
FEE64 module aida10  => 	7
FEE64 module aida11  => 	7
FEE64 module aida12  => 	7


ASIC Clock Timestamp check test result: Passed 12, Failed 0

Database options file size check:


If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status

 Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same

 FEE : aida01 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:14:30 CEST 2021
 FEE : aida02 =>   Options file size is 1014 	 Last changed Fri Apr 16 00:56:20 CEST 2021
 FEE : aida03 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida04 =>   Options file size is 1025 	 Last changed Sat Apr 17 06:07:36 CEST 2021
 FEE : aida05 =>   Options file size is 1025 	 Last changed Fri Apr 16 00:53:25 CEST 2021
 FEE : aida06 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida07 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida08 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:04 CEST 2021
 FEE : aida09 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida10 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:06 CEST 2021
 FEE : aida11 =>   Options file size is 1014 	 Last changed Wed Apr 14 21:52:05 CEST 2021
 FEE : aida12 =>   Options file size is 1025 	 Last changed Wed Apr 14 21:58:54 CEST 2021

System wide checks ok *except*

FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

re-calibration of these FEE modules unsuccessful in FADC align and control.

07:05 FEE64 Temps ok - attachment 12

      good event statistics ok - attachment 13

      detector bias / leakage currents ok - attachment 14

07:10 stats/ucesb/database checks ok

07:30 stats/ucesb/database checks ok