ELOG DESPEC

AIDA GELINA BRIKEN nToF CRIB ISOLDE CIRCE nTOFCapture DESPEC DTAS EDI_PSA 179Ta CARME StellarModelling DCF K40

DESPEC

Fields marked with * are required

Entry time:

Mon Apr 19 06:46:15 2021

Author*:

Subject*:

07:47 
 
&nbsp;&nbsp; &nbsp; 
Clock status test result: Passed 12, Failed 0

Understand status as follows 
Status bit 3 : firmware PLL that creates clocks from external clock not locked 
Status bit 2 : always logic &#39;1&#39;&nbsp; 
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock 
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock 
If all these bits are not set then the operation of the firmware is unreliable

&nbsp; &nbsp;&nbsp; 
FEE64 module aida06 failed 
FEE64 module aida07 failed 
FEE64 module aida10 failed 
Calibration test result: Passed 9, Failed 3

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

&nbsp;

&nbsp;&nbsp; &nbsp; 
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp; Base &nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;Current &nbsp;&nbsp; &nbsp;Difference 
aida01 fault &nbsp;&nbsp; &nbsp; 0x7686 : &nbsp;&nbsp; &nbsp; 0x768a : &nbsp;&nbsp; &nbsp; 4 &nbsp; 
aida02 fault &nbsp;&nbsp; &nbsp; 0x941d : &nbsp;&nbsp; &nbsp; 0x9421 : &nbsp;&nbsp; &nbsp; 4 &nbsp; 
aida03 fault &nbsp;&nbsp; &nbsp; 0x7cd7 : &nbsp;&nbsp; &nbsp; 0x7cdb : &nbsp;&nbsp; &nbsp; 4 &nbsp; 
aida04 fault &nbsp;&nbsp; &nbsp; 0xb86d : &nbsp;&nbsp; &nbsp; 0xb871 : &nbsp;&nbsp; &nbsp; 4 &nbsp; 
aida05 fault &nbsp;&nbsp; &nbsp; 0x1a59 : &nbsp;&nbsp; &nbsp; 0x1a61 : &nbsp;&nbsp; &nbsp; 8 &nbsp; 
aida06 fault &nbsp;&nbsp; &nbsp; 0x4f45 : &nbsp;&nbsp; &nbsp; 0x4f4d : &nbsp;&nbsp; &nbsp; 8 &nbsp; 
aida07 fault &nbsp;&nbsp; &nbsp; 0x3bfc : &nbsp;&nbsp; &nbsp; 0x3c46 : &nbsp;&nbsp; &nbsp; 74 &nbsp; 
aida08 fault &nbsp;&nbsp; &nbsp; 0xc7ce : &nbsp;&nbsp; &nbsp; 0xc7d5 : &nbsp;&nbsp; &nbsp; 7 &nbsp; 
aida09 fault &nbsp;&nbsp; &nbsp; 0xb33b : &nbsp;&nbsp; &nbsp; 0xb33c : &nbsp;&nbsp; &nbsp; 1 &nbsp; 
White Rabbit error counter test result: Passed 3, Failed 9

Understand the status reports as follows:- 
Status bit 3 : White Rabbit decoder detected an error in the received data 
Status bit 2 : Firmware registered WR error, no reload of Timestamp 
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR

&nbsp;

&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp; 
&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp; Base &nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;Current &nbsp;&nbsp; &nbsp;&nbsp;&nbsp; &nbsp;Difference 
aida09 fault &nbsp;&nbsp; &nbsp; 0x1 : &nbsp;&nbsp; &nbsp; 0x2 : &nbsp;&nbsp; &nbsp; 1 &nbsp; 
aida12 fault &nbsp;&nbsp; &nbsp; 0xa : &nbsp;&nbsp; &nbsp; 0x35 : &nbsp;&nbsp; &nbsp; 43 &nbsp; 
FPGA Timestamp error counter test result: Passed 10, Failed 2 
If any of these counts are reported as in error 
The ASIC readout system has detected a timeslip. 
That is the timestamp read from the time FIFO is not younger than the last

&nbsp;

&nbsp;&nbsp; &nbsp; 
Returned 0 0 0 0 0 0 0 0 0 0 0 0 &nbsp; 
Mem(KB) :&nbsp;&nbsp; &nbsp;4&nbsp;&nbsp; &nbsp;8&nbsp;&nbsp; &nbsp;16&nbsp;&nbsp; &nbsp;32&nbsp;&nbsp; &nbsp;64&nbsp;&nbsp; &nbsp;128&nbsp;&nbsp; &nbsp;256&nbsp;&nbsp; &nbsp;512&nbsp;&nbsp; &nbsp;1k&nbsp;&nbsp; &nbsp;2k&nbsp;&nbsp; &nbsp;4k 
aida01 : &nbsp; &nbsp; 19 &nbsp; &nbsp; 18 &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp;7 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 36380 
aida02 : &nbsp; &nbsp; 11 &nbsp; &nbsp; 15 &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;6 &nbsp; : 36692 
aida03 : &nbsp; &nbsp; 15 &nbsp; &nbsp; 11 &nbsp; &nbsp; 13 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 36164 
aida04 : &nbsp; &nbsp; 22 &nbsp; &nbsp; 14 &nbsp; &nbsp; 20 &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 36456 
aida05 : &nbsp; &nbsp; &nbsp;7 &nbsp; &nbsp; &nbsp;7 &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 36132 
aida06 : &nbsp; &nbsp; 21 &nbsp; &nbsp; 14 &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;6 &nbsp; : 37028 
aida07 : &nbsp; &nbsp; 11 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;7 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 37212 
aida08 : &nbsp; &nbsp; 18 &nbsp; &nbsp; 10 &nbsp; &nbsp; &nbsp;7 &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 36072 
aida09 : &nbsp; &nbsp; 24 &nbsp; &nbsp; 23 &nbsp; &nbsp; 18 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 36504 
aida10 : &nbsp; &nbsp; &nbsp;6 &nbsp; &nbsp; 19 &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 35520 
aida11 : &nbsp; &nbsp; 22 &nbsp; &nbsp; 15 &nbsp; &nbsp; &nbsp;7 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 35648 
aida12 : &nbsp; &nbsp; 35 &nbsp; &nbsp; 13 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;6 &nbsp; : 36724

&nbsp;

At 8:30 DAQ and Temperature Scan stopped working properly.

&nbsp;

11:00

It was noted that at the time that AIDA dropped out ~8:30 the beam also dropped out. It seems a bit of a coincidence

TD and MS powercycled the FEEs after the first crash and restarted MIDAS but not the merger. This recovered the FEEs but did not re-establish the links between the FEEs and the Merger.

Another powercycle was performed this time with a full reset of the merger and the links were restored. Upon restoring there was a large amount of noise in across all FEEs on average a 50% increase across all FEEs but in DSSD a factor of 4-6 increase was common.

In the waveforms large 100kHz transiets could be seen - attachment 4

A further power-cycle was performed by OH. The rates following this powercycle are better than just before the power cycle but have not recovered to pre-glitch levels.

The waveforms here are much improved though - attachment 5 and 6

&nbsp;

11:45

FEE64 module aida09 global clocks failed, 6 
&nbsp;Clock status test result: Passed 11, Failed 1

&nbsp;

FEE64 module aida01 failed 
FEE64 module aida02 failed 
FEE64 module aida03 failed 
FEE64 module aida04 failed 
FEE64 module aida05 failed 
FEE64 module aida06 failed 
FEE64 module aida07 failed 
FEE64 module aida08 failed 
FEE64 module aida09 failed 
FEE64 module aida10 failed 
FEE64 module aida11 failed 
FEE64 module aida12 failed 
Calibration test result: Passed 0, Failed 12

If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module

&nbsp;

&nbsp;&nbsp; &nbsp; 
White Rabbit error counter test result: Passed 12, Failed 0

&nbsp;

FPGA Timestamp error counter test result: Passed 12, Failed 0 
If any of these counts are reported as in error 
The ASIC readout system has detected a timeslip. 
That is the timestamp read from the time FIFO is not younger than the last

&nbsp;

&nbsp;&nbsp; &nbsp; 
Returned 0 0 0 0 0 0 0 0 0 0 0 0 &nbsp; 
Mem(KB) :&nbsp;&nbsp; &nbsp;4&nbsp;&nbsp; &nbsp;8&nbsp;&nbsp; &nbsp;16&nbsp;&nbsp; &nbsp;32&nbsp;&nbsp; &nbsp;64&nbsp;&nbsp; &nbsp;128&nbsp;&nbsp; &nbsp;256&nbsp;&nbsp; &nbsp;512&nbsp;&nbsp; &nbsp;1k&nbsp;&nbsp; &nbsp;2k&nbsp;&nbsp; &nbsp;4k 
aida01 : &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 40148 
aida02 : &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 39380 
aida03 : &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 39692 
aida04 : &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 39664 
aida05 : &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;7 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 40160 
aida06 : &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 40192 
aida07 : &nbsp; &nbsp; 23 &nbsp; &nbsp; &nbsp;7 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 39988 
aida08 : &nbsp; &nbsp; 23 &nbsp; &nbsp; 11 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 39860 
aida09 : &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;7 &nbsp; : 40396 
aida10 : &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 39852 
aida11 : &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 39116 
aida12 : &nbsp; &nbsp; &nbsp;5 &nbsp; &nbsp; &nbsp;8 &nbsp; &nbsp; &nbsp;0 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;1 &nbsp; &nbsp; &nbsp;4 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;2 &nbsp; &nbsp; &nbsp;3 &nbsp; &nbsp; &nbsp;7 &nbsp; : 39668

&nbsp;

Encoding: HTML ELCode plain
Suppress Email notification Resubmit as new entry

Attachment 1: