14:11 - Alpha has been running most of morning
Just saw rates in tape spike to 6 MB/s...
Stop output to tape and look:
ASIC check & load... all rates except aida04 back to 0
Start output to tape
Back to ~300 KB/s
R6_49 will be affected by this.
Others seem OK
System wide check failures:
Base Current Difference
aida01 fault 0xb405 : 0xb406 : 1
aida02 fault 0xefc7 : 0xefc8 : 1
aida03 fault 0xdaab : 0xdaac : 1
aida04 fault 0x8f7c : 0x8f7d : 1
aida05 fault 0xb5bd : 0xb5be : 1
aida06 fault 0xeff1 : 0xeff2 : 1
aida07 fault 0x8f57 : 0x8f58 : 1
aida08 fault 0xbef7 : 0xbef8 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x200 : 512
aida13 fault 0x0 : 0x61 : 97
FPGA Timestamp error counter test result: Passed 14, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The FPGA errors seem to come on FEEs with no WR errors or clock errors (including Lock/PLL page)
Temps & Rates OK
Will now stop Alpha run and check noise situation
Run stopped at R6_49
Switching to nostorage (NOTAPE/R2)
Set ASICs to threshold 0xa, Rates OK
Rate to Tape/MBS ca 9 MB/s
No change to situation since yesterday... good sign
During MBS test aida13 rebooted... crash log from ttyUSB5 attached
It was later noticed connector half off in aida05, reseated and after some time got noise levels OK again...
cables on top very sensitive, indicative of issues inside maybe(?). We avoid touching AIDA for now :)
A power cycle of all FEEs was performed after aida13's reboot to make sure things are good.
All systems check OK |