07:47
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c46 : 74
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0x35 : 43
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 18 10 7 1 3 2 4 2 3 6 : 36380
aida02 : 11 15 5 3 2 3 2 3 1 4 6 : 36692
aida03 : 15 11 13 3 4 3 3 3 2 3 6 : 36164
aida04 : 22 14 20 5 1 3 4 3 2 3 6 : 36456
aida05 : 7 7 5 2 1 2 3 4 2 3 6 : 36132
aida06 : 21 14 6 2 3 3 1 2 2 4 6 : 37028
aida07 : 11 8 7 4 2 3 2 2 4 3 6 : 37212
aida08 : 18 10 7 5 3 1 4 3 2 3 6 : 36072
aida09 : 24 23 18 3 2 3 2 4 2 3 6 : 36504
aida10 : 6 19 5 2 2 2 2 3 2 3 6 : 35520
aida11 : 22 15 7 2 1 3 2 3 2 3 6 : 35648
aida12 : 35 13 8 2 5 3 1 3 3 3 6 : 36724
At 8:30 DAQ and Temperature Scan stopped working properly.
11:00
It was noted that at the time that AIDA dropped out ~8:30 the beam also dropped out. It seems a bit of a coincidence
TD and MS powercycled the FEEs after the first crash and restarted MIDAS but not the merger. This recovered the FEEs but did not re-establish the links between the FEEs and the Merger.
Another powercycle was performed this time with a full reset of the merger and the links were restored. Upon restoring there was a large amount of noise in across all FEEs on average a 50% increase across all FEEs but in DSSD a factor of 4-6 increase was common.
In the waveforms large 100kHz transiets could be seen - attachment 4
A further power-cycle was performed by OH. The rates following this powercycle are better than just before the power cycle but have not recovered to pre-glitch levels.
The waveforms here are much improved though - attachment 5 and 6
11:45
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 12, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 2 2 1 2 4 2 4 2 3 7 : 40148
aida02 : 5 4 2 2 1 2 2 3 2 3 7 : 39380
aida03 : 3 2 1 1 1 3 1 4 2 3 7 : 39692
aida04 : 4 2 1 0 1 3 3 3 2 3 7 : 39664
aida05 : 2 7 2 0 2 2 3 4 2 3 7 : 40160
aida06 : 2 3 2 2 2 2 1 3 3 3 7 : 40192
aida07 : 23 7 2 0 0 3 2 4 2 3 7 : 39988
aida08 : 23 11 4 0 1 3 1 4 2 3 7 : 39860
aida09 : 1 3 1 1 2 4 1 3 1 4 7 : 40396
aida10 : 1 3 3 1 1 4 1 2 3 3 7 : 39852
aida11 : 3 0 2 3 1 2 3 2 2 3 7 : 39116
aida12 : 5 8 0 3 1 4 2 3 2 3 7 : 39668
|