| ID |
Date |
Author |
Subject |
|
190
|
Thu Mar 11 23:07:23 2021 |
CA | March 12th 00:00 - 08:00 shift | 00:10 DAQ continues OK - file R30_372
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:17 System wide checks all OK
FEE64 Temperatures ok - attachment 1
Good event statistics ok - attachment 2
detector bias and leakage currents ok - attachment 3
00:20 DESPEC on run 145
00:24 Merger ok ~45M items/s
Tapeserver ok ~15MB/s
01:00 aida07 crashed
recovered, but all FEE64 crash shortly after
called OH - stop DAQ, TapeService, Merger
01:20 all FEE64 powercycled, AIDA restart
01:30 AIDA recovered, DAQ now running - writing to R33
system wide checks ok *except*
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
01:52 FEE64 Temperatures ok - attachment 4
-had to reload a few times to work, but otherwise ok
Good event statistics - attachment 5
-aida05 and aida08 running faster than before
detector bias and leakage currents ok - attachment 6
02:04 writing to file R33_34
Data forwarding to MBS ok
AIDA ASIC settings ok
02:11 beam off
02:13 attempted to recalibrate aida07 and aida09 in FADC Align and Control - calibration still fails
02:32 beam back - writing to file R33_48
04:30 DESPEC having issues with Go4 crashing
ucesb reports AIDA/FATIMA/bplast timewarp events
System wide checks:
Clock error:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Calibration:
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
White Rabbit:
Base Current Difference
aida07 fault 0xcdd1 : 0xcdd2 : 1
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FEE64 Temperatures ok - attachment 7
Good event statistics ok - attachment 8
detector bias and leakage currents ok - attachment 9
04:52 analyzer output for R33_113 - attachment 10
no timewarps
aida06 dead time? (ignore idle time and rates)
05:02 Merger 5M data items/s
TapeServer 17 MB/s
05:15 DESPEC believe issue with bplast TAMEX causing ucesb issues/Go4 crash
They disable bplast and FATIMA TAMEX histograms in their online analysis - ucesb/go4 much more stable now
05:25 rates spectra - attachment 11
05:29 error message in MBS relay terminal - otherwise data forwarding to MBS ok - attachment 12
06:29 system wide checks:
Clock error:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Calibration:
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
White Rabbit:
Base Current Difference
aida05 fault 0x7a56 : 0x7a58 : 2
aida06 fault 0x65bb : 0x65bd : 2
aida07 fault 0xcdd1 : 0xcdd4 : 3
aida08 fault 0x2ab3 : 0x2ab5 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
rest ok
06:32 FEE64 Temperatures ok - attachment 13
Good event statistics ok - attachment 14
detector bias and leakage currents ok - attachment 15
06:41 Merger 5.1M data items/s
TapeServer 17 MB/s
Data forwarding to MBS ok
08:54 restart MBS relay, requested by NH
|
|
197
|
Sat Mar 13 23:08:08 2021 |
CA | 14th March 00:00-08:00 | 00:08 DAQ continues ok - writing to file R46_221
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:09 system wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x82a0 : 0x82a4 : 4
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0x2 : 0x3 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 11 6 2 5 1 3 2 2 3 4 10 : 54492
aida02 : 12 7 3 4 3 3 3 2 3 3 6 : 36440
aida03 : 4 4 2 4 2 3 2 2 3 3 6 : 36048
aida04 : 18 9 4 2 1 3 2 4 1 4 15 : 73936
aida05 : 31 11 8 7 6 4 2 18 7 4 7 : 55220
aida06 : 36 8 18 4 4 2 1 4 3 2 5 : 31088
aida07 : 26 7 4 3 1 2 1 3 3 3 6 : 36224
aida08 : 25 12 5 4 4 3 2 3 3 3 6 : 36884
aida09 : 2 9 4 2 2 2 2 3 3 3 6 : 36432
aida10 : 11 3 4 6 4 3 1 3 3 3 6 : 36548
aida11 : 21 8 5 0 1 3 3 3 2 3 6 : 35748
aida12 : 10 6 6 2 4 4 1 3 3 3 6 : 36600
00:14 FEE64 Temperatures ok - attachment 1
Good event statistics ok - attachment 2
Detector bias/leakage currents ok - attachment 3
00:17 Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
01:30 beam off
01:31 beam back
02:19 system wide checks
same as entry at 00:09, *except*
Base Current Difference
aida07 fault 0x82a0 : 0x82a5 : 5
White Rabbit error counter test result: Passed 11, Failed 1
02:22 FEE64 Temperatures ok - attachment 4
Good event statistics ok - attachment 5
Detector bias/leakage currents ok - attachment 6
02:30 Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
no further bad timestamp errors in NewMerger terminal
02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7
02:50 ... and sure enough lose connection to aida07 after checking stats
DAQ stop, Merger/TapeServer closed
telnet aida07 -> sent reboot command
mounted ok, reset DAQ, TapeService and Merge
restarted DATA forwarding to MBS
all looks to be ok now -> writing to R56
03:09 system wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x2 : 0x5 : 3
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 13 5 3 7 2 2 2 2 3 4 10 : 54508
aida02 : 2 7 3 2 2 3 1 4 3 3 6 : 36784
aida03 : 8 8 1 3 2 3 1 3 3 3 6 : 36304
aida04 : 20 9 2 3 0 3 2 4 1 4 15 : 73880
aida05 : 51 74 70 54 53 49 22 18 7 4 7 : 72188
aida06 : 15 15 3 9 1 2 2 4 3 2 5 : 31044
aida07 : 24 6 2 3 3 4 2 3 2 3 7 : 39888
aida08 : 19 15 1 2 1 3 1 4 3 3 6 : 36820
aida09 : 18 4 2 7 3 4 1 3 3 3 6 : 36648
aida10 : 20 7 1 1 1 3 3 2 3 3 6 : 36216
aida11 : 25 7 5 3 4 2 1 4 2 3 6 : 35916
aida12 : 3 3 2 2 3 2 3 2 3 3 6 : 36164
03:18 FEE64 Temperatures ok - attachment 8
Good event statistics ok - attachment 9
Detector bias/leakage currents ok - attachment 10
Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
03:29 more bad timestamp error messages - attachment 11
03:52 analyser output on R56_35 - attachment 12
no timewarps
03:53 DAQ continues ok, writing to file R56_41
05:18 system wide checks:
same as entry at 03:08, *but* with additional WR decoder status error
Base Current Difference
aida07 fault 0x8c74 : 0x8c77 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
05:19 FEE64 Temperatures ok - attachment 14
Good event statistics ok - attachment 13
Detector bias/leakage currents ok - attachment 15
Merger ok - 4.4M items/s
TapeServer ok - 15 MB/s
05:25 rate spectra - attachment 16
05:33 AIDA implant XY hit patterns - attachments 17/18/19
05:56 BuTIS interface control set to 0x3 at 04.52 UTC
07:06 system wide checks - same error messages as entry at 05:18
07:08 FEE64 Temperatures ok - attachment 20
good event statistics ok - attachment 21
detector bias/leakage currents ok - attachment 22
Merger ok - 4.3M items/s
TapeServer ok - 15 MB/s
07:22 no bad timestamp errors since 02:30 UTC |
|
251
|
Tue Apr 20 07:06:20 2021 |
CA | April 20th 08:00 - 12:00 | 08:09 DAQ crashes at start of shift (!)
again happens at same time as beam being lost
All FEE64 lost connection
Beam downtime for next 3-4 hours
Performed powercycle and reset
08:40 AIDA back up and running, writing to R36
08:42 *all* system wide checks ok
temperatures ok - attachment 1
statistics ok - attachment 2
detector bias/leakage currents ok - attachment 3
08:57 options file size check;
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Tue Apr 20 08:22:14 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
08:59 no error messages in database check terminal
stats ok
09:16 waveforms - attachments 4 & 5
09:30 no error messages in database check terminal
stats ok
10:07 no error messages in database check terminal
stats ok
10:30 aida01, aida02, aida03, aida04, aida07 lost connection, DAQ crashes
powercycle attempted, but FEE modules remain unconnected
TD performs further power cycle
11.12 reboot aida05
DAQ reset/setup
system wide checks
sync asic clocks OK
clock status & fpga timestamp errors OK
WR decoder status
Base Current Difference
aida09 fault 0xb307 : 0xb307 : 0
aida09 : WR status 0x40
aida10 fault 0x3d6c : 0x3d6c : 0
aida10 : WR status 0x40
aida11 fault 0xa610 : 0xa610 : 0
aida11 : WR status 0x40
aida12 fault 0x7c7 : 0x7c7 : 0
aida12 : WR status 0x40
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
NH had access to area, unplugged and replugged HDMI cables from aida09, aida10, aida11, aida12 root to MACB (at both ends)
earlier on, Akash secured HDMI cable to aida07
11:55 AIDA powercycle and reset, DAQ ok now
writing to file NULL/R45
12:00 CB takes over |
|
255
|
Tue Apr 20 23:08:22 2021 |
CA | Wed Apr 21 00:00-08:00 | 00:00 - CA takes over
00:15 - stats/database/ucesb ok - attachment 1
- another burst of bad timestamp error messages in NewMerger terminal - but all FEE64 still ok - attachment 2
- Merger/TapeService ok
00:21 - beam still off while UNILAC having problems
00:26 - beam is back - rate spectra - attachment 3
00:32 - ucesb following beam return - attachment 4
- implant rates reaching highs of ~1kHz
00:35 - Grafana leakage current monitor last 6 hours - attachment 5
01:00 *all* system wide checks ok
stats ok - attachment 6
temperatures ok - attachment 7
detector bias / leakage currents ok - attachment 8
db check ok / ucesb ok
still observe periodic bursts of bad timestamp errors in NewMerger terminal - FEE64s still linked with merger ok
01:30 AIDA crashes just as I make check, most FEE64 lost connection -> powercycle and restart MIDAS
01:53 AIDA reset completed - writing to file NULL/R48
01:56 all system wide checks ok *except*
FPGA error counter
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
ADC calibration
FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
02:01 good event statistics ok - attachment 9
temperatures ok - attachment 10
detector bias / leakage currents ok - attachment 11
db check / ucesb ok
02:20 beam stop
02:21 beam back
02:30 stats/ucesb/db checks all ok
03:00 stats/ucesb/db checks all ok - attachment 12
03:30 stats/ucesb/db checks all ok
04:00 system wide checks ok, except;
ADC calibration
FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
FPGA errors
Base Current Difference
aida12 fault 0x0 : 0x9 : 9
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
04:03 beam off briefly
04:05 good event statistics ok - attachment 13
04:07 beam back
temperatures ok - attachment 14
bias / leakage currents ok - attachment 15
db check / ucesb ok - attachment 16
implant rates in AIDA peaking at ~1.5kHz
04:30 stats/ucesb/db checks all ok
04:48 no beam
04:58 beam back
05:00 stats/ucesb/db checks all ok - attachment 17
05:30 stats/ucesb/db checks all ok
06:00 system wide checks ok, except;
ADC calibration
FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
FPGA errors
Base Current Difference
aida12 fault 0x0 : 0x9 : 9
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
FEE64 Linux Memory Info
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 5 5 3 3 2 3 2 3 2 3 7 : 39628
aida02 : 22 11 3 0 2 3 2 3 2 3 7 : 39648
aida03 : 20 10 1 1 2 2 1 4 2 3 7 : 39760
aida04 : 17 10 2 2 3 3 2 3 2 3 7 : 39732
aida05 : 1 8 3 2 3 2 2 3 3 3 7 : 40564
aida06 : 5 6 2 3 2 2 3 3 2 3 7 : 39748
aida07 : 18 4 7 0 1 3 1 4 2 3 7 : 39832
aida08 : 22 10 3 3 2 2 3 3 2 3 7 : 39864
aida09 : 3 6 2 2 0 3 1 4 2 3 7 : 39708
aida10 : 2 4 3 2 1 2 2 3 2 3 7 : 39384
aida11 : 20 8 3 1 1 3 3 2 2 3 7 : 39328
aida12 : 16 6 3 1 0 3 2 3 2 3 7 : 39488
06:02 good event statistics ok - attachment 18
FEE64 temperatures ok - attachment 19
detector bias / leakage currents ok - attachment 20
06:06 db check and ucesb ok - attachment 21
NewMerger terminal hasn't flagged up any more errors since ~ 03:00
06:30 stats/ucesb/db checks all ok
DESPEC restarts timesorter
07:00 stats/ucesb/db checks all ok
07:30 stats/ucesb/db checks all ok - attachment 22
grafana leakage currents last 1 hour - attachment 23
|
|
259
|
Wed Apr 21 23:05:39 2021 |
CA | 22nd April 00:00 - 08:00 | 00:07 stats/ucesb/DB checks all ok
00:30 all system wide checks ok *except* fpga errors
Base Current Difference
aida12 fault 0x0 : 0x1f : 31
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
statistics ok - attachment 1
temperatures ok - attachment 2
detector bias / leakage currents ok - attachment 3
no error messages in DB terminal & ucesb ok - attachment 4
01:00 stats/ucesb/DB checks all ok
01:30 stats/ucesb/DB checks all ok
01:53 issues with beam dropping in and out - ESR working to stabilise this
02:00 stats/ucesb/DB checks all ok
02:09 beam back in a stable state
02:30 all system wide checks ok *except* fpga errors
Base Current Difference
aida12 fault 0x0 : 0x29 : 41
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
02:33 statistics ok - attachment 5
temperatures ok - attachment 6
detector bias / leakage currents ok - attachment 7
no error messages in DB terminal & ucesb ok - attachment 8
02:45 beam off - potentially serious issue, awaiting update from FRS
03:00 significant issues with UNILAC, beam not expected back for a while
03:05 stats/ucesb/DB checks all ok
03:32 stats/ucesb/DB checks all ok
03:55 beam is back
04:00 stats/ucesb/DB checks all ok
04:30 all system wide checks ok *except* fpga errors
Base Current Difference
aida12 fault 0x0 : 0x2b : 43
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
04:32 statistics ok - attachment 9
temperatures ok - attachment 10
detector bias / leakage currents ok - attachment 11
no error messages in DB terminal & ucesb ok - attachment 12
05:00 stats/ucesb/DB checks all ok
05:30 stats/ucesb/DB checks all ok
06:00 stats/ucesb/DB checks all ok
06:30 all system wide checks ok *except* fpga errors
Base Current Difference
aida12 fault 0x0 : 0x34 : 52
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
06:32 temperatures ok - attachment 13
statistics ok - attachment 14
detector bias / leakage currents ok - attachment 15
no error messages in DB terminal & ucesb ok - attachment 16
07:00 stats/ucesb/DB checks all ok
07:30 stats/ucesb/DB checks all ok
07:59 stats/ucesb/DB checks all ok
|
|
316
|
Fri May 14 07:05:08 2021 |
CA | Friday 14th May 08:00 - 16:00 shift | 08:00 CA takes over
08:28 Problem at UNILAC - no beam
09:30 all system wide checks ok *except*
all FEE64 fail ADC calibration (no waveforms)
WR decoder status:
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc2 : 10
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp errors:
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
09:36 FEE64 Temperatures ok - attachment 1
Detector bias / leakage currents ok - attachment 2
Statistics - attachments 3-8
09:50 beam is back - writing to file R4_265
09:56 FRS adjusting degrader settings (S4) - temporarily remove to check counts in scintillator vs AIDA
10:10 FRS increase degrader thickness
10.28 all histograms, stats and merger stats zero'd
10.50 all system wide checks ok *except*
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc2 : 10
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 3 2 2 2 2 2 2 4 3 3 6 : 36860
aida02 : 7 3 5 1 2 5 2 4 3 3 6 : 37284
aida03 : 7 5 3 1 4 4 2 3 3 3 6 : 36756
aida04 : 3 1 20 4 2 3 3 5 1 3 6 : 36052
aida05 : 19 25 26 6 2 2 3 2 2 3 6 : 35828
aida06 : 1 3 16 1 3 4 1 5 1 3 6 : 35580
aida07 : 6 4 1 2 3 3 2 3 3 3 6 : 36552
aida08 : 8 5 2 3 2 3 2 2 2 4 6 : 37064
aida09 : 9 7 1 2 2 4 2 3 3 3 6 : 36652
aida10 : 2 5 3 1 1 4 2 3 3 3 6 : 36544
aida11 : 16 4 4 2 1 2 2 3 3 3 6 : 36384
aida12 : 2 1 1 3 1 2 2 3 3 3 6 : 36288
aida13 : 0 1 1 2 0 4 3 2 3 3 6 : 36184
aida14 : 6 3 2 0 2 3 2 3 3 3 6 : 36432
aida15 : 25 8 2 1 2 3 1 2 2 4 6 : 36836
aida16 : 3 5 1 2 2 2 3 2 3 3 6 : 36100
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1026 Last changed Thu May 13 05:49:42 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1025 Last changed Wed May 05 12:15:54 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1025 Last changed Fri May 07 19:40:34 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
10.50 DAQ continues S496/R4_304
11:18 FEE64 Temperatures ok - attachment 9
Detector bias & leakage currents ok - attachment 10
statistics - attachments 11-16
15:00 FEE64 avg. CPU usage (%)
1 52
2 55
3 55
4 60
5 93
6 50
7 53
8 55
9 55
10 51
11 50
12 67
13 56
14 55
15 55
16 58
15:08 all system wide checks ok *except*
all FEE64 fail ADC calibration (no waveform)
WR decoder status:
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dc3 : 11
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA errors:
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
15:10 Temperatures ok - attachment 17
Detector bias / leakage current ok - attachment 18
Statistics - attachments 18 -24 |
|
326
|
Tue May 18 23:11:40 2021 |
CA | Wednesday May 19th 00:00 - 08:00 | 00:00 CA takes over
LS has performed usual checks, see previous Elog entry - all ok
00:30 stats ok
R14_470 deadtime at 4.4%
02:10 system wide checks ok except:
all ADC fail calibration (expected, no waveforms)
WR decoder status
Base Current Difference
aida05 fault 0x500 : 0x52f : 47
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
02:12 FEE64 Temps ok - attachment 1
Stats ok - attachment 2
detector bias / leakage currents ok - attachment 3
dead-times ok - R14_497 max 4% - attachment 4
03:27 DAQ continues ok - writing to file R14_517
04:07 system wide checks ok except:
all ADC fail calibration (expected, no waveforms)
WR decoder status
Base Current Difference
aida05 fault 0x500 : 0x537 : 55
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FEE64 Temps ok - attachment 6
Stats ok - attachment 5
detector bias / leakage currents ok - attachment 7
dead-times ok - R14_526 max 4% - attachment 8
05:27 bad timestamp error in merger terminal - attachment 9
06:05 all system wide checks ok except;
all ADC fail calibration (expected, no waveforms)
WR decoder status
Base Current Difference
aida05 fault 0x500 : 0x53d : 61
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
06:07 FEE64 Temps ok - attachment 10
Stats ok - attachment 11
detector bias / leakage currents ok - attachment 12
dead-times ok - R14_558 max 3.6% - attachment 13
07:01 another bad timestamp error in merger terminal - attachment 14 |
|
328
|
Wed May 19 09:06:58 2021 |
CA | May 16th 00:00 - 08:00 | 16th May 2021 - 00:00 - 08:00 Shift
Author: CA
00:15 System wide checks;
FEE64 module aida06 global clocks failed, 6
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 14, Failed 2
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x714c : 0x7154 : 8
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 10 3 1 2 3 2 3 2 3 7 : 39660
aida02 : 29 5 3 1 1 3 2 3 2 3 7 : 39596
aida03 : 15 7 1 1 2 3 2 3 2 3 7 : 39588
aida04 : 44 26 19 5 2 4 3 3 2 2 7 : 38608
aida05 : 20 10 3 0 1 4 2 2 3 3 7 : 40208
aida06 : 27 6 2 1 2 4 2 4 2 3 7 : 40284
aida07 : 25 5 3 1 2 3 2 3 2 3 7 : 39644
aida08 : 23 8 2 3 1 4 2 3 2 3 7 : 39772
aida09 : 23 10 3 2 3 4 1 3 2 3 7 : 39644
aida10 : 18 10 3 2 3 2 2 2 2 4 7 : 41160
aida11 : 16 7 3 2 1 2 2 3 2 3 7 : 39464
aida12 : 21 4 7 1 3 3 3 3 2 3 7 : 40004
aida13 : 25 6 3 2 1 3 1 4 2 3 7 : 39876
aida14 : 20 8 10 1 1 2 2 2 2 4 7 : 41104
aida15 : 26 6 3 3 0 4 3 2 2 3 7 : 39464
aida16 : 9 8 2 4 1 3 1 4 2 3 7 : 39876
00:30 Stats ok - 210516_0030_stats.png
Temps ok - 210516_0030_temps.png
Bias ok - 210516_0030_bias.png
00:38 analyzer output R7_178 - Deadtime at 7.3% - 210516_R7_178_analysis.txt
02:21 system wide checks as above, except;
Base Current Difference
aida05 fault 0x714c : 0x7159 : 13
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x7 : 7
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
02:28 Stats ok - 210516_0230_stats.png
Temps ok - 210516_0230_temps.png
Bias ok - 210516_0230_bias.png
02:33 analyzer output R7_216 - Deadtime at 6.5% - 210516_R7_216_analysis.txt
04:36 system wide checks same as above, except;
Base Current Difference
aida05 fault 0x714c : 0x715c : 16
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x9 : 9
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
04:37 Stats ok - 210516_0430_stats.png
Temps ok - 210516_0430_temps.png
Bias ok - 210516_0430_bias.png
04:44 analyzer output R7_260 - Deadtime at 6.9% - 210516_R7_260_analysis.txt
06:38 system wide checks same as before, except;
Base Current Difference
aida01 fault 0xf294 : 0xf296 : 2
aida02 fault 0xd8ec : 0xd8ee : 2
aida03 fault 0xf001 : 0xf003 : 2
aida04 fault 0xd992 : 0xd994 : 2
aida05 fault 0x714c : 0x7161 : 21
aida06 fault 0x5a49 : 0x5a4a : 1
aida07 fault 0x5aca : 0x5acb : 1
aida08 fault 0xb92e : 0xb92f : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
06:40 Stats ok - 210516_0640_stats.png
Temps ok - 210516_0640_temps.png
Bias ok - 210516_0640_bias.png
06:45 analyzer output R7_298 - Deadtime at 5.8% - 210516_R7_298_analysis.txt |
|
336
|
Thu May 20 07:03:02 2021 |
CA | Thursday May 20th 08:00 - 16:00 shift | 08:00 CA takes over
08:04 DAQ continues ok - writing to file R14_989
09:30 all system wide checks ok *except* WR decoder status;
Base Current Difference
aida05 fault 0x500 : 0x56a : 106
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
09:34 temperatures ok - attachment 1
stats ok - attachment 2
bias/leakage currents ok - attachment 3
R14_1016 analysis - max deadtime ~4.3% - attachment 4
11:30 all system wide checks ok *except* WR decoder status;
Base Current Difference
aida05 fault 0x500 : 0x56b : 107
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
11:34 temperatures ok - attachment 5
stats ok - attachment 6
bias/leakage currents ok - attachment 7
R14_1052 analysis - max deadtime ~3.6% - attachment 8
13:05 beam optimisation currently taking place
DAQ continues ok and writing to file R14_1085
13:17 no beam
13:28 all system wide checks ok, WR decoder status difference same as entry at 11:30.
13:31 temperatures ok - attachment 9
stats ok - attachment 10
bias/leakage currents ok - attachment 11
R14_1085 analysis (beam off) - max deadtime ~1.34% - attachment 12
13:37 beam is back - rate spectra - attachment 13
DAQ ok - writing to file R14_1088
13:43 analysis of R14_1088 - max deadtime back up to ~4.5% - attachment 14
14:21 beam off - issue with UNILAC
DESPEC also report issues with their DAQ
AIDA continues to forward data to MBS ok - lower rates while beam off
14.30 1.8.H spectra - attachments 16-18
1.8.L spectra - attachments 19-22
aida01 pulser peak width 99 ch FWHM
Rate spectra - attachment 23
Grafana - DSSSD bias & leakage currents past 7 days - attachment 24
DSSSD bias & leakage currents - attachment 25
Merger/Tape Server/Merger stats - attachment 26
14.41 All histograms & stats zero'd
14.55 Lost Activity Monitors - attachments 27, 28 & 35
Stats - ADC data items, disc, good wave, WR 28-47, pause, correlation scaler
15:00 beam is back
15:05 AIDA writing to file R14_1108
ADC data items - attachment 36
15:27 system wide checks ok *except* WR decoder status
Base Current Difference
aida05 fault 0x500 : 0x56c : 108
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
15:30 temperatures ok - attachment 37
stats ok - attachment 38
bias/leakage currents ok - attachment 39
R14_1113 analysis (beam off) - max deadtime ~4% - attachment 40 |
|
342
|
Fri May 21 15:03:48 2021 |
CA | Friday 21st May 16:00 - 00:00 shift | 16:00 CA takes over
DSSD1 n+n sides 0x20 on ASICs 1-3 and 0x64 on ASIC 4
DSSD2 n+n sides 0x1b on ASICs 1-3 and 0x64 on ASIC 4
PULSER SETTINGS
---------------------------
Pulse is ON
Positive Tail Pulse
Trigger Source is Internal Clock
Trigger Threshold is 3.5
Amplitude : 2.0 Volts
Rep Rate : 2.0 hZ
Delay : 250.0 ns
Fall Time : 1 ms
Attenuation : 1
Display is : Volts
Equivalent keV is : 200.0
Ramp Start at 0.01 Volts
Ramp Stop at 9.99 Volts
Ramp Start at 1.0 keV
Ramp Stop at 999.0 keV
Ramp Time is 60 seconds
# Ramp Cycles is 1
16:14 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x570 : 112
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
16:17 temperatures ok - attachment 1
statistics ok - attachment 2
detector bias / leakage currents ok - attachment 3
analysis of R14_1625 - FEE7 deadtime at 4.5%
18:06 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x571 : 113
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
18:07 temperatures ok - attachment 4
statistics ok - attachment 5
detector bias / leakage currents ok - attachment 6
analysis of R14_1662 - FEE7 deadtime at 4.3%
19:07 DAQ continues ok - writing to file R14_1682
20:07 system wide checks all ok *except*
Base Current Difference
aida05 fault 0x500 : 0x572 : 114
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
20:10 temperatures ok - attachment 7
statistics ok - attachment 8
detector bias / leakage currents ok - attachment 9
analysis of R14_1703 - max deadtime at 4.0% - attachment 10
20:35 rate spectra - attachment 11
HEC spectra - attachment 12 & 13
20:36 all histograms zeroed
20:37 DAQ continues ok - writing to file R14_1712
20:45 /media/1e121361-83d3-4825-b6ae-9700b07e0ca7 at 94% use.
~ 12 hours until 100% at 10 MB/s data rate, not accounting for compression of older files
21:12 no beam - writing to file R14_1723
21:17 beam is back
22:07 system wide checks all ok - WR decoder fault same as entry at 20:07
22:08 temperatures ok - attachment 14
statistics ok - attachment 15
detector bias / leakage currents ok - attachment 16
analysis of R14_1740 - max deadtime at 4.2% - attachment 17
|
|
616
|
Sun Apr 28 15:09:34 2024 |
Betool Alayed | 16:0-00:00 28 April 2024 | 16:00 aida04 not producing data
power cycle all FEE64s to recover DAQ
16.20 analysis data file R21_668
max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%
no timewarps
HEC data item rate 1.9kHz
|
|
619
|
Sun Apr 28 16:26:47 2024 |
Betool Alayed | 16:00-00:00 Sunday 28 April | 19:24 pm screenshots
and
17:18 pm screenshots |
|
620
|
Sun Apr 28 20:35:57 2024 |
Betool Alayed | 16:0-00:00 28 April 2024 | 21:30 screenshots
and
23:24 screenshots |
|
657
|
Wed Jun 12 15:41:07 2024 |
Betool Alayed | 16:00-00:00 Wed 12 Jun 24 | 5pm full checks:
DSSSD bias & leakage current ok - attachment # 1-2
FEE64 temperatures ok - attachment # 3
ADC data item stats - attachment # 4
Merger ok - Attachement # 5
Tape service - attachement # 6
7pm full checks:
DSSSD bias & leakage current ok - attachment # 7-8
FEE64 temperatures ok - attachment # 9
ADC data item stats - attachment # 10
Merger ok - Attachement # 11
Tape service - attachement # 12
ucesb - attachment # 13
9pm full checks:
DSSSD bias & leakage current ok - attachment# 14-15
FEE64 temperatures ok - attachment # 16
ADC data item stats - attachment # 17
Merger ok - Attachement # 18
Tape service - attachement # 19
ucesb - attachment # 20
11pm full checks:
DSSSD bias & leakage current ok - attachment# 21-22
FEE64 temperatures ok - attachment # 23
ADC data item stats - attachment # 24
Merger ok - Attachement # 25
Tape service - attachement # 26
ucesb - attachment # 27 |
|
237
|
Sun Apr 18 00:57:36 2021 |
BA, MA | Sanday 18 April 00.00-08.00 | 02:01 Beam has stopped at 01:37 and returend at 01:43 for few min and then stopped again and not knowen how it will take until it is back
AIDA scalers attached 1
statistic attached 2
temretuer attached 3
bias attached 4
Clock check ok
ADC check :
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a55 : 3
aida06 fault 0x4f3e : 0x4f41 : 3
aida07 fault 0x3bcd : 0x3bea : 29
aida08 fault 0xc7c7 : 0xc7ca : 3
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x9 : 9
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 32 8 3 1 0 3 1 3 3 3 6 : 36240
aida02 : 11 7 9 3 1 2 2 4 2 3 6 : 35988
aida03 : 32 4 13 5 5 4 3 3 2 3 6 : 36432
aida04 : 26 7 6 1 2 3 2 4 2 3 6 : 36128
aida05 : 18 8 8 7 4 2 2 2 2 4 6 : 37352
aida06 : 24 11 3 1 2 5 1 3 3 3 6 : 36616
aida07 : 14 5 3 3 3 2 3 2 4 3 6 : 37296
aida08 : 21 7 1 5 0 1 2 3 3 3 6 : 36284
aida09 : 0 7 1 1 1 3 2 2 3 3 6 : 35880
aida10 : 22 5 3 4 1 2 2 2 3 3 6 : 35952
aida11 : 4 3 1 1 2 2 3 3 2 3 6 : 35544
aida12 : 22 10 5 4 4 4 1 3 3 3 6 : 36728
02:19 beam is back
03:58 The beam has not been stable yet
The rate reach 1.5 kHz, they will contact FRS team to lower the intensity of the beam
AIDA scalers attached 8
statistic attached 7
temretuer attached 6
bias attached 5
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a55 : 3
aida06 fault 0x4f3e : 0x4f41 : 3
aida07 fault 0x3bcd : 0x3beb : 30
aida08 fault 0xc7c7 : 0xc7ca : 3
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x9 : 9
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 21 8 6 2 1 2 1 3 3 3 6 : 36212
aida02 : 24 14 14 2 1 2 2 4 2 3 6 : 36144
aida03 : 31 5 10 5 5 4 2 3 2 3 6 : 36132
aida04 : 12 11 12 2 2 2 3 4 2 3 6 : 36360
aida05 : 23 8 5 7 4 2 2 2 2 4 6 : 37324
aida06 : 21 14 14 1 3 5 1 3 3 3 6 : 36868
aida07 : 15 9 5 0 3 2 3 2 4 3 6 : 37268
aida08 : 21 11 10 3 0 1 2 3 3 3 6 : 36396
aida09 : 5 7 5 1 1 3 1 3 3 3 6 : 36220
aida10 : 15 13 12 3 2 1 2 2 3 3 6 : 36036
aida11 : 13 10 1 0 2 2 2 4 2 3 6 : 35860
aida12 : 29 5 4 5 5 3 1 3 3 3 6 : 36668
05: 07 The rate was a bout 1500 and 2000, we contacted Oscar he said (if it's just bursts it should be ok), so they decided to do nothing.
06:26
AIDA scalers attached 9
statistic attached 10
temretuer attached 11
bias attached 12
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0x7685 : 0x7686 : 1
aida02 fault 0x941c : 0x941d : 1
aida03 fault 0x7cd6 : 0x7cd7 : 1
aida04 fault 0xb86c : 0xb86d : 1
aida05 fault 0x1a52 : 0x1a59 : 7
aida06 fault 0x4f3e : 0x4f45 : 7
aida07 fault 0x3bcd : 0x3bf9 : 44
aida08 fault 0xc7c7 : 0xc7ce : 7
aida09 fault 0xb33a : 0xb33b : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0xa : 10
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 17 12 6 2 1 2 1 3 3 3 6 : 36228
aida02 : 19 11 12 3 2 2 2 4 2 3 6 : 36164
aida03 : 25 8 12 5 4 4 3 3 2 3 6 : 36356
aida04 : 22 19 11 3 1 2 3 4 2 3 6 : 36416
aida05 : 35 6 5 6 3 2 2 2 3 3 6 : 36236
aida06 : 12 11 11 2 3 4 1 3 3 3 6 : 36664
aida07 : 18 6 2 1 3 2 3 2 3 3 6 : 36216
aida08 : 27 10 8 3 0 1 2 3 3 3 6 : 36380
aida09 : 18 6 6 1 0 2 2 2 3 3 6 : 35832
aida10 : 3 14 10 3 1 1 2 3 3 3 6 : 36412
aida11 : 1 3 2 0 2 2 2 4 2 3 6 : 35772
aida12 : 0 5 6 3 3 4 1 3 3 3 6 : 36520
07:57 The beam stopped and they said there is a water leak !
|
|
434
|
Thu May 12 21:43:49 2022 |
BA, MA | Rates |
Statistic check (screenshot attached).
Temperatures OK (screenshot attached).
Bias and leakage currents OK (same screenshot as temps).
|
|
435
|
Thu May 12 22:49:39 2022 |
BA, MA | Low Rates | The rate is very low in all aida, not sure . |
|
451
|
Sat May 14 15:35:24 2022 |
BA, AA | Saturday 14 May | |
|
452
|
Sat May 14 17:34:40 2022 |
BA, AA | Saturday 14 May | FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 13, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 14, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
|
|
453
|
Sat May 14 19:39:13 2022 |
BA, AA | Saturday 14 May | FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 13, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida02 failed
FEE64 module aida06 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida13 failed
Calibration test result: Passed 9, Failed 5
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 14, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 14, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last |
|