 Mon Apr 19 01:08:40 2021, MA, Monday 19th April 00:00-08:00 8x
|
03:09 General check
Rates, temptature, voltages are OK attached 1, 2, 3, 4
****Clock Ckeck******
OK
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******ADC check ******
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
****** White Rabbit check******
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c2c : 48
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
*****FPGA check ******
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0x10 : 6
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
*****Memory check*****
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 6 12 11 7 2 3 2 4 2 3 6 : 36360
aida02 : 21 15 7 3 2 3 1 4 2 3 6 : 35996
aida03 : 9 10 10 2 4 3 3 3 2 3 6 : 36052
aida04 : 17 17 17 4 0 2 3 4 2 3 6 : 36444
aida05 : 25 7 3 1 1 1 2 3 2 4 6 : 37292
aida06 : 29 15 7 2 3 4 1 3 3 3 6 : 36700
aida07 : 17 17 6 5 2 3 2 3 3 3 6 : 36812
aida08 : 22 6 10 5 3 1 3 4 2 3 6 : 36360
aida09 : 18 28 14 4 2 3 3 3 2 3 6 : 36232
aida10 : 26 16 4 3 3 1 3 4 2 3 6 : 36296
aida11 : 16 10 4 0 3 3 1 4 2 3 6 : 35856
aida12 : 19 14 4 2 4 4 1 3 3 3 6 : 36668
04:15 General Check
Rates, Tempratures, Voltages are ok, attached 5,6,7,8
******Clock Check*******
OK
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******ADC******
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******White Rabbit check******
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c39 : 61
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
******FPGA Check******
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0x11 : 7
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
****** Memorey check******
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 21 10 7 1 3 2 4 2 3 6 : 36436
aida02 : 28 21 3 3 1 3 1 4 2 3 6 : 35944
aida03 : 15 8 11 3 3 3 3 3 2 3 6 : 36044
aida04 : 25 19 14 4 1 3 4 3 2 3 6 : 36380
aida05 : 12 6 5 1 0 2 2 3 2 4 6 : 37328
aida06 : 29 21 4 2 3 4 1 3 3 3 6 : 36700
aida07 : 19 14 7 6 2 3 2 2 3 3 6 : 36332
aida08 : 11 8 8 6 3 1 3 4 2 3 6 : 36332
aida09 : 16 25 11 3 1 2 2 4 2 3 6 : 36184
aida10 : 14 13 4 2 3 2 2 4 2 3 6 : 36064
aida11 : 4 9 4 0 3 3 1 4 2 3 6 : 35800
aida12 : 38 17 4 2 4 4 1 3 3 3 6 : 36768
|
|
Wed Jun 22 15:09:37 2022, MA, Wednesday 22 June 16:00-00:00 12x
|
16:00 Take over from Magda mrning shift
Beam was off of sometime for tunning to Mercurey setting.
17:00 attached 1-3
19:00 attached 4-6
21:00 attached 7-9
23:30 10-12
|
|
Mon Jun 27 17:03:09 2022, MA, Monday 27th June 16:00-00:00 6x
|
16:00 Took over the shift from OH no beam yet.
18:00 Still no beam yet.
Statistics, Temperature, Current are checked and attached 1-3
system wide checks same as last updated in the previoues shift.
22:00 The beam is back but not taking data yet! FRS team doing some checkings
Statistics, Temperature, Current are checked and attached 4-6
system wide checks same as last updated in the previoues shift.
23:30 beam is back and taking data |
|
Thu Mar 12 23:04:24 2020, LS, DK, Friday 13th 00:00 to 08:00 shift 20x
|
Shift file starting R9_639
system wide checks okay
FEE temperatures okay (attachment 1)
leakage currents normal and recorded to spreadsheet (attachment 2)
good event stats are normal (attachment 3)
merger running at 3.5 M items/sec
tape running at 26 MB/sec
rates histogram (attachment 4)
low energy histograms (attachment 5 and 6)
high energy histograms (attachment 7 and 8)
02.02
system wide checks okay
FEE temperatures okay (attachment 9)
leakage currents normal and recorded to spreadsheet (attachment 10)
good event stats are normal (attachment 11)
merger running at 3.5 M items/sec
tape running at 27 MB/sec
04.09
system wide checks okay
FEE temperatures okay (attachment 12)
leakage currents normal and recorded to spreadsheet (attachment 13)
good event stats are normal (attachment 14)
merger running at 3 M items/sec
tape running at 27 MB/sec
writing to file R9_840
06.07
system wide checks okay
FEE temperatures okay (attachment 15)
leakage currents normal and recorded to spreadsheet (attachment 16)
good event stats are normal (attachment 17)
merger running at 3 M items/sec
tape running at 27 MB/sec
07.59
system wide checks okay
FEE temperatures okay (attachment 18)
leakage currents normal and recorded to spreadsheet (attachment 19)
good event stats are normal (attachment 20)
merger running at 3 M items/sec
tape running at 26 MB/sec
as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space |
|
Mon Mar 9 13:50:08 2020, LS, CA, DK, 09/03/2020 system checks 13x
|
14:52 Attachment 1 shows low energy spectra. AIDA06 shows strange baseline, possible double-hit?
15:06 Systems checks fine.
Master clock failed (no master clock)
all checks passed
memory checks all around 38k as normal
Attachment 2: Temperature check all normal except aida01 virtex temp slightly over 65 degrees
Attachment 3: Good event stats seem normal except downstream SSD seems to be high around 200k
Attachment 4: leakage currents seem normal at expected values
15:16 Merger working for all FEEs
tape server is on but no storage
data rate is around 25 MB per second (high but consistent with the high stats of downstream SSD)
ASIC controls checked and are all OK
15.48 low energy pulsar peak FWHM
fee FHWM
1 89.08
2 132.06
3 70.73
4 81.84
5 62.86
6 79.92
7 122.13
8 73.31
9 198.39
10 115.53
11 100.55
12 144.04
ALL FEEs in downstream SSD are high
16:21 included low energy pulsar peaks (Attachments 5 & 6) and waveforms for all FEEs (Attachments 7 & 8)
16:25 included rates, note not to common scale in y (attachment 9)
ASIC settings file 2019Dec19-16.19.51 but some others are on 2019Oct31-13.24.23
Did not seem completely reproducible?
We carefully checked all the individual FEE and ASIC settings, and they are all the same (except shaping reference), with NO 0xad
Now we made sure they are all using 2019Dec19-16.19.51 (which we had saved first on a good one)
19:15 Update
There is a safety interlock box in S4 to monitor humidity, dew point, temperature, etc for safety.
It had a loose solder connection, and when someone moved it out of the way, the interlock was tripped.
After some debugging, the interlock is now in a more stable condition, but a lot of power was cut from AIDA systems
One unusual thing was that one of the RPi systems got a full /var/messages (or similar) and ate all the available disk space
This meant that ssh -X (to, e.g., activate putty) could not work, giving a permission denial error (ssh connection was allowed, but not X11 forwarding)
Now we have brought back up the AIDA systems and should run through the checks
Most system checks look good (except some calibration errors)
Temps were running high until we reloaded the ASIC settings, then the temperatures began to cool
Bias and leak currents attached as #10
Temps as #11
Stats as #12
19:50 Started writing to disk so that we can do an implantation depth profile. Will analyse near online using AIDASort.
21:45 Stopped writing to file -> Runs where AIDA thickness were investigated corresponds to R3_62 to R3_72
A rough method using the rates histogram was used to judge roughly when depositing in each histogram - see attachment 12 (Stopping in 2)
Can match to the FRS files to work out degrader thickness with timestamps (File stopped at Mon 09 Mar 2020 09:44:51 PM CET
21:51 AIDA running to no storage again |
|
Fri Mar 12 10:09:19 2021, LS, CA, Friday 12th March 11.00- 17x
|
11.00(Germany) System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment1)
Spectra rates (attachment2)
FEE temps (attachment3)
Leakage currents, written to google sheet (attachment4)
Merger~ 4.9M items/s
Tapeserver ~17MB/s
In MBS control terminal, connection has been closed intentionally since this morning (file S452f160),
AIDA has been taken out of the timesorter due to the high data rate, buffers were full
AIDA cannot be seen in ucesb or Go4.
13.00 System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment5)
Spectra rates (attachment6)
FEE temps (attachment7)
Leakage currents, written to google sheet (attachment8)
Merger~ 5.1M items/s
Tapeserver ~18MB/s
No timestamp related errors this shift
13.37 AIDA MBS control restarted R33_388
System wide checks same as previous time
13.54 beam stopped for access, file R33_396
13.57 seen recent batch of bad timestamp errors in new merger terminal (attachment9) should be around R33_397
Analysed R33_395, 396, 397, 398 no timewarps
14.29 beam back R33_415
14.54 see more bad timestamps in new merger terminal (attachment10)
14.55 System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment11)
Spectra rates (attachment12)
FEE temps (attachment13)
Leakage currents, written to google sheet (attachment14)
Merger~ 5.0M items/s
Tapeserver ~17MB/s
16.05 AIDA included back into timesorter, AIDA scalers now seen in ucesb, file R33_463
16.15 CA takes over until 18:00
17:11 System wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Base Current Difference
aida07 fault 0xcdd6 : 0xcdd7 : 1
White Rabbit error counter test result: Passed 11, Failed 1
17:13 FEE64 temps ok - attachment 15
Statistics ok - attachment 16
bias and leakage currents ok - attachment 17 |
|
Mon May 17 19:05:23 2021, LS, Monday 17th May 20.00-24.00 21x
|
20.30
Checked stats, seem okay (Fig 1)
Ran R14_99 through analyser, dead times all good (Fig 2)
Merger rate 1.8M items/s
Tape server 6.4MB/s
21.00
Checked stats seem okay (Fig 3)
Ran R14_106 through analyser, dead times all good (Fig 4)
21.30
Checked stats seem okay (Fig 5)
Ran R14_112 through analyser, dead times all good (Fig 6)
22.00
Stats okay (Fig 7)
FEE temps okay (Fig 8)
Rate spectra (Fig 9)
Leakage currents (Fig 10)
System wide checks:
Clock status all passed
White rabbit 1 fail:
Base Current Difference
aida05 fault 0x4da : 0x4e7 : 13
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp all passed
Merger rate 2.4M items/s
Tape server 6.3MB/s
Ran R14_119 through analyser, dead times all good (Fig 11)
22.30
Checked stats seem okay (Fig 12)
Ran R14_123 through analyser, dead times all good (Fig 13)
23.00
Checked stats seem okay (Fig 14)
Ran R14_129 through analyser, dead times all good (Fig 15)
23.30
Checked stats seem okay (Fig 16)
Ran R14_135 through analyser, dead times all good (Fig 17)
24.00
Stats okay (Fig 18)
FEE temps okay (Fig 19)
Leakage currents (Fig 20)
System wide checks:
Clock status all passed
White rabbit 1 fail:
Base Current Difference
aida05 fault 0x4da : 0x4eb : 17
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp all passed
Merger rate 2.5M items/s
Tape server 6.5MB/s
Ran R14_142 through analyser, dead times all good (Fig 21) |
|
Tue May 18 18:47:25 2021, LS, Tuesday 18th May 20:00 - 24:00 24x
|
19.50
Stats okay (Fig 1)
FEE temps okay (Fig 2)
Bias okay (Fig 3)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x520 : 32
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~1.8M items/s
Tape server ~7MB/s
Analysed R14_399, deadtimes are okay (Fig 4)
20.30
Stats okay (Fig 5)
Analysed R14_407, deadtimes are okay (Fig 6)
21.00
Stats okay (Fig 7)
Analysed R14_415, deadtimes are okay (Fig 8)
21.30
Stats okay (Fig 9)
Analysed R14_422, deadtimes are okay (Fig 10)
22.00
Stats okay (Fig 11)
FEE temps okay (Fig 12)
Bias okay (Fig 13)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x524 : 36
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~2.3M items/s
Tape server ~8MB/s
Analysed R14_430, deadtimes are okay (Fig 14)
22.30
Stats okay (Fig 15)
Analysed R14_438, deadtimes are okay (Fig 16)
FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5%
23.00
Stats okay (Fig 17)
Analysed R14_445, deadtimes are okay (Fig 18)
23.30
Stats okay (Fig 19)
Analysed R14_454, deadtimes are okay (Fig 20)
24.00
Stats okay (Fig 21)
FEE temps okay (Fig 22)
Bias okay (Fig 23)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x529 : 41
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
The difference has increased over time (not sure if an issue)
Merger rate ~3M items/s
Tape server ~8MB/s
Analysed R14_462, deadtimes are okay (Fig 24) |
|
Wed May 19 18:50:01 2021, LS, Wednesday 19th May 20:00 to 24:00 21x
|
20.00
Stats okay (Fig 1)
Analysed R14_762 deadtimes are okay (Fig 2)
20.30
Stats okay (Fig 3)
Analysed R14_771 deadtimes are okay (Fig 4)
21.00
Stats okay (Fig 5)
FEE temps okay (Fig 6)
Bias (Fig 7)
Rate spectra (Fig 8)
System wide checks okay except white rabbit 1 fail:
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~2.8M items/s
Tape server ~10MB/s
Analysed R14_781 deadtimes are okay (Fig 9)
21.30
Stats okay (Fig 10)
Analysed R14_789 deadtimes are okay (Fig 11)
22.00
Stats okay (Fig 12)
Analysed R14_799 deadtimes are okay (Fig 13)
22.30
Stats okay (Fig 14)
Analysed R14_807 deadtimes are okay (Fig 15)
23.00
Stats okay (Fig 16)
FEE temps okay (Fig 17)
Bias (Fig 18)
System wide checks okay except white rabbit 1 fail:
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~3M items/s
Tape server ~10MB/s
Analysed R14_819 deadtimes are okay (Fig 19)
23.30
Stats okay (Fig 20)
Analysed R14_827 deadtimes are okay (Fig 21) |
|
Sat Apr 17 07:28:02 2021, LPG, TD, Saturday 08:00 - 12:00 9x
|
08:30 CEST
HV and leakage currents: elog:233/1
Detector rates: elog:233/2
Temperatures: elog:233/3
WR status:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
FPGA status:
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
09:20 CEST
Stats: ok!
DB: No faults found
ucesb: ok!
09:40 CEST
Beam will be stopped in order to increase intensity. Expected to be around 3-4 hours.
For now, we still get implants when Beam spill is on. Seems to be fluctuating as they play around.
10:00 CEST
Still getting beam, it is fluctuating in intensity
Stats: ok!
DB: No faults found
ucesb: ok!
HV and leakage currents: elog:233/4
Detector rates: elog:233/5
Temperatures: elog:233/6
Clock check: ok!
ADC calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
WR check: ok!
FPGA check:
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory check: ok!
10:40 CEST
Stats: ok!
DB: No faults found
ucesb: ok!
11:00 CEST
Stats: ok!
DB: No faults found
ucesb: ok!
11:30 CEST
Stats: ok!
DB: No faults found
ucesb: ok!
12:00 BST
Stats: ok!
DB: No faults found
ucesb: ok!
HV and leakage currents: elog:233/7
Detector rates: elog:233/8
Temperatures: elog:233/9
Clock check: ok!
ADC calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
WR check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bce : 1
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
Memory check: ok! |
|
Tue Apr 20 14:56:38 2021, LPG, Tuesday 20th 16:00-20:00 15x
|
16:00
New shift, ready to go! Beam stopped just as I started... Waiting for update.
Beam back at 16:08, ucesb scalers shown in elog:253/1
Database ok!
Stats ok!
ucesb ok!
Temperatures ok! elog:253/2
Statistics ok! elog:253/3
Leakage currents ok! Steadily rising, day/night effect? - elog:253/4 and elog:253/5
16:21
Online histograms zeroed in MIDAS. Same done in MBS DAQ.
16:45
Database ok!
Stats ok!
ucesb ok!
System wide checks:
Clock ok!
ADC 1 failed:
FEE64 module aida06 failed
WR ok!
FPGA ok!
Memory ok!
17:15
Database ok!
Stats ok!
ucesb ok!
17:45
Database ok!
Stats ok!
ucesb ok!
18:15
Database ok!
Stats ok!
ucesb ok!
System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!
MIDAS Rates ok! elog:253/6
Temperatures ok! elog:253/7
Leakage currents ok, but still slowly rising! Topped out at ~20:30 yesterday elog:253/8
18:42
Message in the Merger terminal window (elog:253/9)
MERGE Actor (15671): Working with 0 from 12 data sources
19:05
Database ok!
Stats ok!
ucesb ok!
19:30
Some MIDAS spectra in elog:253/10 and elog:253/11
Database ok!
Stats ok!
ucesb ok!
19:50
Database ok!
Stats ok!
ucesb ok! elog:253/12
System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!
MIDAS Rates ok! elog:253/13
Temperatures ok! elog:253/14
Leakage currents ok, now plateaued! elog:253/15
19:55 - Handing over to James  |
|
Thu Apr 22 07:06:37 2021, LPG, Thursday 22nd April: 08:00-12:00 8x
|
08:01 - Taking over from Corrigan...
ucesb ok!
DB ok!
Stats ok!
08:30
ucesb ok!
DB ok!
Stats ok!
09:15
ucesb ok! elog:260/4
DB ok!
Stats ok!
Leakage currents ok! elog:260/1
Temperatures ok! elog:260/2
MIDAS Rates ok! elog:260/3
System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x40 : 64
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!
09:45
ucesb ok!
DB ok!
Stats ok!
10:30
ucesb ok!
DB ok!
Stats ok!
11:00
ucesb ok! elog:260/5
DB ok!
Stats ok!
Leakage currents ok! elog:260/6
Temperatures ok! elog:260/7
MIDAS Rates ok! elog:260/8
System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x42 : 66
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!
11:30
ucesb ok! elog:260/5
DB ok!
Stats ok!
11:55
ucesb ok!
DB ok!
Stats ok! |
|
Wed Apr 21 15:01:14 2021, LJW, MA, Wednesday 21st April 16:00-20:00 13x
|
16:30
All System checks okay, except for:
FPGA Timestamp error:
Base Current Difference
aida12 fault 0x0 : 0x1b : 27
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Temperature Checks: See attachment 1
Statistics Checks: See attachment 2
Bias & Leakage Currents: See attachment 3
18:09
All system checks okay, except for:
FPGA Timestamp error:
Base Current Difference
aida12 fault 0x0 : 0x1d : 29
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Temperature checks: See attachment 4
Statistics Checks: See attachment 5
Bias & Leakage Currents: See attachment 6
19:40
All system checks okay, except for:
FPGA Timestamp error:
Base Current Difference
aida12 fault 0x0 : 0x1d : 29
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Temperature checks: See attachment 7
Statistics Checks: See attachment 8
Bias & Leakage Currents: See attachment 9
*Beam being tuned to another experiment (R3B) - all DSSD implants fluctuating back and forth to 0 Hz for around the last hour or so*
22:30
General check
Rates, Voltages, Temperatures, Ucesb are attached 10,11,12,13
Its been noticed that aida04 temp. is incresing to about 65 (red), still below 70.
the system wide check are all ok except FPGA
Base Current Difference
aida12 fault 0x0 : 0x1d : 29
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
|
|
Mon Apr 19 16:58:29 2021, LJW, Monday 19th April 16:00-20:00 Shift 6x
|
System Checks @ ~18:10 :
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0xd052 : 0xd05a : 8
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x23 : 35
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 6 4 0 1 5 2 3 2 3 6 : 35636
aida02 : 27 3 2 0 2 3 2 3 3 3 6 : 36516
aida03 : 33 9 2 1 1 3 2 3 3 3 6 : 36556
aida04 : 11 6 5 4 2 4 3 3 3 3 6 : 37036
aida05 : 15 8 7 4 3 4 4 2 3 3 6 : 36908
aida06 : 13 9 7 3 3 4 2 3 3 3 6 : 36876
aida07 : 1 3 3 2 2 3 1 3 3 3 6 : 36236
aida08 : 29 5 2 1 1 2 2 3 3 3 6 : 36380
aida09 : 3 8 2 0 2 3 2 2 2 4 6 : 36972
aida10 : 9 5 3 3 1 4 2 4 2 3 6 : 36124
aida11 : 5 7 3 2 2 2 2 4 2 3 6 : 35900
aida12 : 24 6 3 1 2 3 2 2 3 3 6 : 36064
System Checks @ ~19:55
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xdb9a : 0xdb9d : 3
aida06 fault 0xb74f : 0xb752 : 3
aida07 fault 0xd052 : 0xd05e : 12
aida08 fault 0xe4c1 : 0xe4c4 : 3
aida09 fault 0xaf4b : 0xaf4d : 2
White Rabbit error counter test result: Passed 7, Failed 5
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x27 : 39
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 18 10 2 2 1 5 2 3 2 3 6 : 35768
aida02 : 21 9 2 1 2 2 2 3 3 3 6 : 36444
aida03 : 13 5 6 1 2 2 2 3 3 3 6 : 36444
aida04 : 30 11 4 3 2 4 3 3 3 3 6 : 37104
aida05 : 10 9 8 4 4 4 4 2 3 3 6 : 36976
aida06 : 22 17 10 3 2 3 1 4 3 3 6 : 37088
aida07 : 20 8 4 1 2 4 1 2 3 3 6 : 35952
aida08 : 14 4 3 0 1 2 2 3 3 3 6 : 36296
aida09 : 24 8 4 0 1 3 3 3 3 3 6 : 36768
aida10 : 19 8 4 1 1 4 2 3 3 3 6 : 36652
aida11 : 6 6 1 2 3 1 2 4 2 3 6 : 35800
aida12 : 15 7 1 1 2 4 1 2 3 3 6 : 35876 |
|
Fri Apr 23 07:09:57 2021, LJW, Friday 23rd April 08:00-12:00 9x
|
08:30
System Checks
Clock Status error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 1
Temperatures - See attachment 2
Bias & Leakage Currents - See attachment 3
10:02
System Checks:
Clock Status Error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration Error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status Error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 4
Temperatures - See attachment 5
Bias & Leakage Currents - See attachment 6
10:45
System Checks
Clock Status Error:
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration check Error:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Decoder Status Error:
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error:
Showed larger error message begining 'Got the error Server Internal Error
while trying to obtain /AIDA/Check/Check.tml.'
Statistics - See attacnment 7
Temperatures - See attachment 8
Bias & Leakage Currents - See attachment 9
|
|
Mon May 17 15:14:35 2021, LJW, Monday 17th May 2021 16:00-20:00 9x
|
16:25
Informed at shift handover that there has been no beam.
Statistics - See attachment 1
Temperatures - See attachment 2
Bias & Leakage currents - See attachment 3
System Wide checks - All passed, EXCEPT:
ADC Calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit decoder status:
Base Current Difference
aida05 fault 0x4da : 0x4dd : 3
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
18:00
Usecb scalars now showing DSSD implant rates.
Statistics - See attachment 4
Temperatures - See attachment 5
Bias & Leakage currents - See attachment 6
System Wide checks - All passed, EXCEPT:
ADC Calibration
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White rabbit decoder status
Base Current Difference
aida05 fault 0x4da : 0x4e1 : 7
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WRx
19:45
Usecb scalars have continued to show DSSD implant rates since the last recorded elog check.
Statistics - See attachment 7
Temperatures - See attachment 8
Bias & Leakage currents - See attachment 9
System Wide checks - All passed, EXCEPT:
ADC Calibration:
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
FEE64 module aida13 failed
FEE64 module aida14 failed
FEE64 module aida15 failed
FEE64 module aida16 failed
Calibration test result: Passed 0, Failed 16
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit decoder status:
Base Current Difference
aida05 fault 0x4da : 0x4e1 : 7
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
19:50 - Just as completing checks, notced usesb rates not showing DSSD implants again for several minutes
19:55 - Beam now back again - DSSD implants showing
|
|
Sat Apr 17 11:27:38 2021, JS, TD, Saturday 17th April 12:00-16:00 10x
|
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bce : 1
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
13:55 CEST
Statistics : ok elog:234/4
Temp : ok elog:234/5
Bias : ok elog:234/6
ucesb : ok
DB: No faults found
ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and
Control browser page to rerun calibration for that module
White Rabbit Check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bcf : 2
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
14:05 no beam
14:10 beam back
14:15 no beam, beam current being optimised, going to thicker degrader when beam returns
14:37 CEST
Statistics : ok
Temp : ok
Bias : ok (ch3 now over 6uA)
ucesb : ok
DB: No faults found
15:03 CEST
Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found
15:31 CEST
Statistics : ok
Temp : ok
Bias : ok
ucesb : ok
DB: No faults found
15:50
Statistics : ok elog:234/7
Temp : ok elog:234/8
Bias : ok elog:234/9
ucesb : ok elog:234/10
DB: No faults found
ADC Calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit Check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bcf : 2
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last |
|
Tue Apr 20 19:00:00 2021, JS, Tuesday 20th 20:00-00:00 12x
|
20:00
Taking over from Liam, all systems good.
20:30
DB No faults.
Stats good
ucesb ok
20:58
DB No faults.
Stats good
Temps good
ucesb ok
21:31
DB No faults.
Stats good
Temps good
ucesb ok
22:02
DB No faults.
Stats good elog:254/3
Temps good elog:254/4
ucesb ok elog:254/1
Current ok elog:254/2
ADC Calibration check:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
22:06 ucesb - DAQ Error - Onsite team checking daq crash
Tape server icon stopped spinning
Stats Rates 0, Counter aida01 zeroed but others normal. elog:254/5
Called Tom. Tried resetting in midas, no luck. Full reboot of DAQ and midas required.
According to log at 22:07 aida01 reset
Data taking stopped at 22:04 file R45_193
22:50
After a full power cycle all appear to be up and running well. Starting full checks.
23:00
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
Check FPGA is having a page error
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k
4k
aida01 : 4 8 2 2 1 3 1 4 2 3 7 :
39792
aida02 : 13 9 2 4 2 3 2 3 2 3 7 :
39708
aida03 : 19 5 4 3 1 3 1 4 2 3 7 :
39892
aida04 : 14 5 2 2 0 5 2 3 2 3 7 :
39744
aida05 : 16 11 3 2 2 4 3 2 1 4 7 :
40584
aida06 : 15 9 5 2 2 2 1 4 2 3 7 :
39828
aida07 : 15 9 3 3 2 2 1 4 2 3 7 :
39828
aida08 : 21 4 3 4 2 2 1 4 2 3 7 :
39844
aida09 : 11 6 1 2 3 4 2 3 2 3 7 :
39788
aida10 : 0 6 4 2 0 3 2 3 2 3 7 :
39472
aida11 : 16 7 2 0 1 3 2 3 2 3 7 :
39512
aida12 : 3 7 4 1 3 2 2 3 2 3 7 :
39524
DB No faults.
Stats good elog:254/6
Temps good elog:254/7 aida01 had been high during reboot issue, >70, but now is back down
Current ok elog:254/8
ucesb ok
23:25
DB No faults.
Stats good
Temps good
ucesb ok
23:42 No Beam
23:48 Beam Back
23:57
DB No faults.
Stats good elog:254/11
Temps good elog:254/12
ucesb ok elog:254/9
Current ok elog:254/10
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
*FPGA check page error*
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 9 5 2 1 2 1 4 2 3 7 : 39812
aida02 : 13 9 5 2 2 2 1 4 2 3 7 : 39820
aida03 : 21 8 3 2 2 2 1 4 2 3 7 : 39812
aida04 : 13 4 3 1 1 5 2 3 2 3 7 : 39780
aida05 : 24 5 3 2 2 4 2 3 2 3 7 : 39800
aida06 : 16 5 5 2 2 2 1 4 2 3 7 : 39800
aida07 : 22 6 2 2 1 3 1 4 2 3 7 : 39848
aida08 : 21 8 2 3 2 2 1 4 2 3 7 : 39828
aida09 : 18 7 1 1 3 4 2 3 2 3 7 : 39792
aida10 : 19 3 3 1 0 3 2 3 2 3 7 : 39476
aida11 : 17 4 4 0 1 3 2 3 2 3 7 : 39524
aida12 : 13 4 3 2 3 1 2 3 2 3 7 : 39428
00:04 Handing over to Corrigan |
|
Thu Apr 22 11:03:57 2021, JS, Thursday 22nd April: 12:00-16:00 12x
|
12:00 Taking over from Liam, all is well.
12:10 Beam dropped out for a minute, implantation rate is down a bit, peaking around 300 Hz
12:19 Beam stopped (from GSI log "Run stopped because of a call from HKR stating they have a problematic device which they
need to check and will need apprx. 15 min.")
12:30
ucesb ok (no beam)
DB No faults
Stats ok.
12:34 In order to fix some ongoing FRS problem no beam expected for at least an hour.
13:00
We are taking intermittent beam while they work on the FRS
DB No faults
ucesb ok elog:261/1
Current ok elog:261/2
Stats ok elog:261/3
Temps ok elog:261/4
Clock status test result: Passed 12, Failed 0
Calibration test result: Passed 12, Failed 0
White Rabbit error counter test result: Passed 12, Failed 0
Base Current Difference
aida12 fault 0x0 : 0x46 : 70
FPGA Timestamp error counter test result: Passed 11, Failed 1
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 6 5 3 2 2 2 3 2 3 7 : 39580
aida02 : 21 7 4 2 2 3 2 3 2 3 7 : 39692
aida03 : 20 10 3 3 0 3 2 3 2 3 7 : 39600
aida04 : 8 7 4 5 2 4 3 2 2 3 7 : 39608
aida05 : 17 14 1 2 3 3 4 2 2 3 7 : 39748
aida06 : 17 10 2 3 2 2 4 2 2 3 7 : 39572
aida07 : 27 10 8 7 2 3 3 2 2 3 7 : 39708
aida08 : 9 6 2 3 2 1 3 3 2 3 7 : 39636
aida09 : 1 2 1 4 0 3 3 2 2 3 7 : 39204
aida10 : 5 8 4 4 2 2 1 3 2 3 7 : 39316
aida11 : 6 4 5 3 2 3 2 2 2 3 7 : 39144
aida12 : 15 5 8 1 0 4 1 3 2 3 7 : 39428
13:32
ucesb ok
DB No faults
Stats ok
14:01
ucesb ok
DB No faults
Stats ok
14.30 analysis of file /TapeData/S460/R50_528 - attachment 5
cat /var/log/messages | grep /MIDAS - attachments 6 & 7
Grafana DSSSD bias & leakage currents for last 7 days - attachment 8
14:55
DB No faults
ucesb ok
Current ok elog:261/9
Stats ok elog:261/10
Temps ok elog:261/11
FPGA Timestamp Check Error:
Base Current Difference
aida12 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 11, Failed 1
15:30
ucesb ok
DB No faults
Stats ok
15:59
ucesb ok elog:261/12
DB No faults
Stats ok |
|
Wed May 12 18:53:38 2021, JS, Wed 12 20:00-00:00
|
19:51 Shift change
Stats ok
ucesb ok, Max implants ~1500 Hz
20:30
Stats ok
ucesb ok, Max implants ~1000 Hz
current ok (maybe leveling out)
20:51
The online people are starting a new sort and are going to change some things with degraders I think. We are on R1_340.
20:59
Check
Clock status test result: Passed 16, Failed 0
Calibration test result: Passed 16, Failed 0
WR:
Base Current Difference
aida05 fault 0x36ca : 0x36ce : 4
White Rabbit error counter test result: Passed 15, Failed 1
FPGA time:
Base Current Difference
aida13 fault 0xa : 0xb6 : 172
FPGA Timestamp error counter test result: Passed 15, Failed 1
--- both errors above same module as earlier shift
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 3 2 0 1 3 3 2 3 3 6 : 36092
aida02 : 6 4 4 2 2 3 1 3 3 3 6 : 36280
aida03 : 6 3 2 0 0 3 3 2 3 3 6 : 36048
aida04 : 10 3 1 1 3 3 1 3 3 3 6 : 36272
aida05 : 19 8 2 0 2 3 1 3 3 3 6 : 36268
aida06 : 2 2 0 2 3 4 2 2 3 3 6 : 36120
aida07 : 6 7 3 0 3 4 3 2 3 3 6 : 36416
aida08 : 7 5 4 0 3 2 2 3 3 3 6 : 36420
aida09 : 21 6 3 2 2 4 2 3 2 3 6 : 35700
aida10 : 13 3 1 1 3 3 1 4 2 3 6 : 35772
aida11 : 25 6 4 2 1 4 2 4 2 3 6 : 36180
aida12 : 0 4 0 3 2 3 3 3 2 3 6 : 35712
aida13 : 18 5 2 3 2 3 2 4 2 3 6 : 36080
aida14 : 2 1 1 1 2 3 2 4 2 3 6 : 35904
aida15 : 0 1 1 3 0 4 2 2 3 3 6 : 35960
aida16 : 4 5 3 2 3 4 1 3 2 3 6 : 35432
ucesb ok elog:309/1
current ok elog:309/2
Stats ok elog:309/3
temps ok elog:309/4
21:36
ucesb ok
Stats ok
22:01
ucesb ok 1500 Hz peak implant
Stats ok
temps ok
bias ok - earlier rise seems to have stopped, 0.2uA fluctuations on values.
22:20 Nic noted that they were seeing fewer y-strips events (x agreed with FRS rate).
Dead time was suggested as a problem. Oscar investigating, good event rate in midas doesn't show significant dead time.
Looking at the difference in FEEs3 & 4, which should have the same scaler rate but don't, suggestion data lost in the pause and resume.
Oscar is going to raise the threshold on one of the y FEEs as a test
Increased aida4 slow comparator threshold changed
There is some confusion as it seems to be dead time but apart rates shown look low enough, dead time is possibly masking true rate?
Results of test, need thresholds 100keV on p+n and 320keV n+n to get a roughly 1:1 between FEE3 and FEE4 scaler
23:40 FRS magnet issue, possibly power, no beam
00:03 bias ok elog:309/5 has come down because no beam
handing over to muneerah |
|