 Sat May 14 01:52:03 2022, ML, 2h-Shift Checks 
|
AIDA stats ok
Leakage current ok
Temperatires ok
grafana ok
ucesb rates dropped a bit in all scalers so beam intensity must have dropped a bit.
System wide check done and same results as earlier:
aida09 fails clock (1), aida02 06 09 10 13 fail ADC calibration, all pass WR |
|
Sun May 15 15:51:49 2022, ML, Sunday 15th May 16:00-0:00 12x
|
Status at 16:45 (CET)
The expreiment continues to run smoothly.
AIDA Stats look ok - Attachment 1
AIDA Temperature ok - Attachment 2
AIDA Leakage current: ok - Attachment 3
System wide check:
Clock: 13 passed, 1 failed (aida09)
ADC Calibration: 9 passed, 5 fialed (aida2,6,9,10,13)
White rabbit decoder: 14 passed, 0 failed.
FPGA timestamp: 14 passed, 0 failed.
Status at 18:45 (CET):
All rates ok on ucesb.
AIDA Stats ok - Attachment 4
AIDA Temperature ok - Attachment 5
AIDA Leakage current: increase slightly - Attachment 6
System wide check: No changes, same as above
Status at 20:45 (CET):
All rates ok on ucesb.
AIDA Stats ok - Attachment 7
AIDA Temperature ok - Attachment 8
AIDA Leakage current: increase slightly - Attachment 9
System wide check: No changes, same as above
Status at 22:45 (CET):
All rates ok on ucesb.
AIDA Stats ok - Attachment 10
AIDA Temperature ok - Attachment 11
AIDA Leakage current: increase slightly - Attachment 12
System wide check: No changes, same as above |
|
Sat Mar 14 23:09:28 2020, MB, SJ, MR, Sunday 15th March 0:00-8:00 16x
|
AIDA ELog filled out by DESPEC night shift members Matthew Brunet (MB), Shaheen Jazrawi (SJ), and Matthias Rudiger (MR).
0:05-0:08 Beam operators altering settings, beam off/sporadic during this period.
0:46
All system wide checks passed
Leak currents lower than previous measurement - attachment #1
Implants look as previously - attachment #2
Stats about where they have been - attachment #3
FEE temperatures fine - attachment #4
Merger fine, 3 Mega items / sec
TapeServer writing about 25 MB/s
2:46
All system wide checks passed
Leak currents similar to previous - attachment #5
Implants look as previously - attachment #6
Stats about where they have been - attachment #7
FEE temperatures fine - attachment #8
Merger fine, 3 Mega items / sec
TapeServer writing about 27 MB/s
4:49
All system wide checks passed
Leak currents similar to previous - attachment #9
Implants look as previously - attachment #10
Stats good - attachment #11
FEE temperatures fine - attachment #12
Merger fine, 3 Mega items / sec
TapeServer writing about 27 MB/s
6:49
All system wide checks passed
Leak currents no significant change - attachment #13
Implants look as previously - attachment #14
Stats good - attachment #15
FEE temperatures fine - attachment #16
Merger fine, 3 Mega items / sec
TapeServer writing about 25 MB/s |
|
Fri Apr 23 13:13:46 2021, MA, TD, Friday 23 April 12.00-16.00 6x
|
14.15 System wide checks
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp errors - reports 'server internal error'
14.16 DSSSD bias & leakage currents OK - attachment 1
FEE64 temps OK - attachment 2
good events stats - attachment 3
14.22 DAQ continues file S460/R51_205
beam intensity 2.2e+09/s
15:53 system check
Rates, Temperatures, Voltages are ok and attached 4,5,6
Clock check
FEE64 module aida06 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC check
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida12 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit
Base Current Difference
aida06 fault 0x679f : 0x67a2 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA
Loge page error!
Memory check
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 23 6 2 1 2 3 2 4 2 3 6 : 36044
aida02 : 15 3 2 2 0 3 2 3 2 3 4 : 27188
aida03 : 29 4 0 3 2 4 2 2 3 3 6 : 36212
aida04 : 4 6 4 4 2 3 3 3 3 3 6 : 36864
aida05 : 19 5 4 2 3 3 1 3 3 3 6 : 36404
aida06 : 24 9 5 2 3 3 3 2 3 3 6 : 36472
aida07 : 20 4 9 0 2 3 2 2 3 3 6 : 36096
aida08 : 22 9 4 3 1 3 1 3 3 3 6 : 36352
aida09 : 20 7 3 2 2 3 3 2 3 3 6 : 36344
aida10 : 25 9 9 5 2 3 2 3 3 3 6 : 36828
aida11 : 24 11 2 0 1 3 1 3 3 3 6 : 36248
aida12 : 26 5 4 2 1 3 2 2 3 3 6 : 36048 |
|
Tue Mar 5 09:10:01 2024, MA, JB, TD, Tuesday 5 March
|
Old ribbon cable assemblies - attachments 1 2
Length of cable from base of AIDA snout assembly to Samtec FFSD ribbon cable connectors
p+n junction side 10-11cm & 25.5cm
n+n Ohmic side 11cm & 25cm
Distance from Delron base of AIDA snout assembly to Samtec CLP connectors ( Kapton PCBs ) 51.5cm & 52.5cm
Manufacture of new ribbon cable assemblies - attachment 3
8x 2x 29" Samtec FFSD 34-way cables
8x 2x 23" Samtec FFSD 34-way cables
17.52 from Helena
"These are the detector positions within the snout:
Total snout length of 573 mm (380 mm bottom stage + 193 top stage), relative to the 'black flange' of the AIDA frame
Upstream AIDA @ 513 mm
Downstream AIDA @ 523 mm
This leaves 5cm space for the Downstream bPlast AND the BB7 layer together. I believe this should be enough space - can you all confirm for AIDA/bPlast/BB7 if this is
agreed upon? It would be great to get a fast response so that we are ready for mounting tomorrow. If we need more space we could think of shifting everything upstream by
10mm.
The upstream bPlast will need to go first such that the upstream AIDA is at 513mm. I don't have the measurements to hand to give the position." |
|
Wed May 12 02:07:59 2021, MA, BA, Wed 12 May 00:00-08:00 9x
|
03:12 System Check
attachment 1 : Current
attachment 2 : Temperature
attachment 3 : Rates
Clock status test result: Passed 16, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 16, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x36ca : 0x36cb : 1
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida13 fault 0xa : 0xf : 5
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 22 5 7 2 1 2 2 3 3 3 6 : 36464
aida02 : 9 8 3 3 1 4 1 2 4 3 6 : 36916
aida03 : 5 2 6 2 0 3 2 3 3 3 6 : 36420
aida04 : 6 5 2 3 4 4 2 3 3 3 6 : 36800
aida05 : 17 6 6 2 2 4 1 3 2 4 6 : 37524
aida06 : 7 12 3 4 3 3 1 3 3 3 6 : 36460
aida07 : 17 11 5 1 3 3 3 2 3 3 6 : 36428
aida08 : 3 5 0 1 4 2 2 4 2 3 6 : 35924
aida09 : 27 6 4 2 0 2 2 2 3 3 6 : 35868
aida10 : 16 11 8 0 2 2 1 3 2 4 6 : 37272
aida11 : 15 2 2 3 1 4 3 4 2 3 6 : 36364
aida12 : 1 6 4 3 1 3 2 4 2 3 6 : 35988
aida13 : 22 14 10 2 4 2 2 4 2 3 6 : 36264
aida14 : 26 10 4 3 2 1 1 3 3 3 6 : 36184
aida15 : 14 2 3 2 2 4 1 2 3 3 6 : 35896
aida16 : 7 5 4 0 2 4 2 3 2 3 6 : 35588
06:10 DSSD1 rate high!
Attached 4
Called OH and he woke up to fix it :)
It was a problem with one of the ASIC according to what he says:
One of the ASICs HEC was running crazy!. Forced the ASICs to check their settings which brings them to back into line see attachment 9. This done by around 06:35
07:03 System check
attachment 5 Spectrum rate
attachment 6 Voltages
attachment 7 Rates
attachment 8 Temperature
System wide clock are all ok except
White rabbit
Base Current Difference
aida05 fault 0x36ca : 0x36cb : 1
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA
Base Current Difference
aida13 fault 0xa : 0x14 : 10
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
|
|
Mon Apr 19 01:08:40 2021, MA, Monday 19th April 00:00-08:00 8x
|
03:09 General check
Rates, temptature, voltages are OK attached 1, 2, 3, 4
****Clock Ckeck******
OK
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******ADC check ******
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
****** White Rabbit check******
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c2c : 48
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
*****FPGA check ******
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0x10 : 6
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
*****Memory check*****
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 6 12 11 7 2 3 2 4 2 3 6 : 36360
aida02 : 21 15 7 3 2 3 1 4 2 3 6 : 35996
aida03 : 9 10 10 2 4 3 3 3 2 3 6 : 36052
aida04 : 17 17 17 4 0 2 3 4 2 3 6 : 36444
aida05 : 25 7 3 1 1 1 2 3 2 4 6 : 37292
aida06 : 29 15 7 2 3 4 1 3 3 3 6 : 36700
aida07 : 17 17 6 5 2 3 2 3 3 3 6 : 36812
aida08 : 22 6 10 5 3 1 3 4 2 3 6 : 36360
aida09 : 18 28 14 4 2 3 3 3 2 3 6 : 36232
aida10 : 26 16 4 3 3 1 3 4 2 3 6 : 36296
aida11 : 16 10 4 0 3 3 1 4 2 3 6 : 35856
aida12 : 19 14 4 2 4 4 1 3 3 3 6 : 36668
04:15 General Check
Rates, Tempratures, Voltages are ok, attached 5,6,7,8
******Clock Check*******
OK
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******ADC******
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
******White Rabbit check******
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c39 : 61
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
******FPGA Check******
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0x11 : 7
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
****** Memorey check******
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 21 10 7 1 3 2 4 2 3 6 : 36436
aida02 : 28 21 3 3 1 3 1 4 2 3 6 : 35944
aida03 : 15 8 11 3 3 3 3 3 2 3 6 : 36044
aida04 : 25 19 14 4 1 3 4 3 2 3 6 : 36380
aida05 : 12 6 5 1 0 2 2 3 2 4 6 : 37328
aida06 : 29 21 4 2 3 4 1 3 3 3 6 : 36700
aida07 : 19 14 7 6 2 3 2 2 3 3 6 : 36332
aida08 : 11 8 8 6 3 1 3 4 2 3 6 : 36332
aida09 : 16 25 11 3 1 2 2 4 2 3 6 : 36184
aida10 : 14 13 4 2 3 2 2 4 2 3 6 : 36064
aida11 : 4 9 4 0 3 3 1 4 2 3 6 : 35800
aida12 : 38 17 4 2 4 4 1 3 3 3 6 : 36768
|
|
Wed Jun 22 15:09:37 2022, MA, Wednesday 22 June 16:00-00:00 12x
|
16:00 Take over from Magda mrning shift
Beam was off of sometime for tunning to Mercurey setting.
17:00 attached 1-3
19:00 attached 4-6
21:00 attached 7-9
23:30 10-12
|
|
Mon Jun 27 17:03:09 2022, MA, Monday 27th June 16:00-00:00 6x
|
16:00 Took over the shift from OH no beam yet.
18:00 Still no beam yet.
Statistics, Temperature, Current are checked and attached 1-3
system wide checks same as last updated in the previoues shift.
22:00 The beam is back but not taking data yet! FRS team doing some checkings
Statistics, Temperature, Current are checked and attached 4-6
system wide checks same as last updated in the previoues shift.
23:30 beam is back and taking data |
|
Thu Mar 12 23:04:24 2020, LS, DK, Friday 13th 00:00 to 08:00 shift 20x
|
Shift file starting R9_639
system wide checks okay
FEE temperatures okay (attachment 1)
leakage currents normal and recorded to spreadsheet (attachment 2)
good event stats are normal (attachment 3)
merger running at 3.5 M items/sec
tape running at 26 MB/sec
rates histogram (attachment 4)
low energy histograms (attachment 5 and 6)
high energy histograms (attachment 7 and 8)
02.02
system wide checks okay
FEE temperatures okay (attachment 9)
leakage currents normal and recorded to spreadsheet (attachment 10)
good event stats are normal (attachment 11)
merger running at 3.5 M items/sec
tape running at 27 MB/sec
04.09
system wide checks okay
FEE temperatures okay (attachment 12)
leakage currents normal and recorded to spreadsheet (attachment 13)
good event stats are normal (attachment 14)
merger running at 3 M items/sec
tape running at 27 MB/sec
writing to file R9_840
06.07
system wide checks okay
FEE temperatures okay (attachment 15)
leakage currents normal and recorded to spreadsheet (attachment 16)
good event stats are normal (attachment 17)
merger running at 3 M items/sec
tape running at 27 MB/sec
07.59
system wide checks okay
FEE temperatures okay (attachment 18)
leakage currents normal and recorded to spreadsheet (attachment 19)
good event stats are normal (attachment 20)
merger running at 3 M items/sec
tape running at 26 MB/sec
as of 08.00 just finished R9_1021
gzip is going through the R6_8* files
2.1TB left of space |
|
Mon Mar 9 13:50:08 2020, LS, CA, DK, 09/03/2020 system checks 13x
|
14:52 Attachment 1 shows low energy spectra. AIDA06 shows strange baseline, possible double-hit?
15:06 Systems checks fine.
Master clock failed (no master clock)
all checks passed
memory checks all around 38k as normal
Attachment 2: Temperature check all normal except aida01 virtex temp slightly over 65 degrees
Attachment 3: Good event stats seem normal except downstream SSD seems to be high around 200k
Attachment 4: leakage currents seem normal at expected values
15:16 Merger working for all FEEs
tape server is on but no storage
data rate is around 25 MB per second (high but consistent with the high stats of downstream SSD)
ASIC controls checked and are all OK
15.48 low energy pulsar peak FWHM
fee FHWM
1 89.08
2 132.06
3 70.73
4 81.84
5 62.86
6 79.92
7 122.13
8 73.31
9 198.39
10 115.53
11 100.55
12 144.04
ALL FEEs in downstream SSD are high
16:21 included low energy pulsar peaks (Attachments 5 & 6) and waveforms for all FEEs (Attachments 7 & 8)
16:25 included rates, note not to common scale in y (attachment 9)
ASIC settings file 2019Dec19-16.19.51 but some others are on 2019Oct31-13.24.23
Did not seem completely reproducible?
We carefully checked all the individual FEE and ASIC settings, and they are all the same (except shaping reference), with NO 0xad
Now we made sure they are all using 2019Dec19-16.19.51 (which we had saved first on a good one)
19:15 Update
There is a safety interlock box in S4 to monitor humidity, dew point, temperature, etc for safety.
It had a loose solder connection, and when someone moved it out of the way, the interlock was tripped.
After some debugging, the interlock is now in a more stable condition, but a lot of power was cut from AIDA systems
One unusual thing was that one of the RPi systems got a full /var/messages (or similar) and ate all the available disk space
This meant that ssh -X (to, e.g., activate putty) could not work, giving a permission denial error (ssh connection was allowed, but not X11 forwarding)
Now we have brought back up the AIDA systems and should run through the checks
Most system checks look good (except some calibration errors)
Temps were running high until we reloaded the ASIC settings, then the temperatures began to cool
Bias and leak currents attached as #10
Temps as #11
Stats as #12
19:50 Started writing to disk so that we can do an implantation depth profile. Will analyse near online using AIDASort.
21:45 Stopped writing to file -> Runs where AIDA thickness were investigated corresponds to R3_62 to R3_72
A rough method using the rates histogram was used to judge roughly when depositing in each histogram - see attachment 12 (Stopping in 2)
Can match to the FRS files to work out degrader thickness with timestamps (File stopped at Mon 09 Mar 2020 09:44:51 PM CET
21:51 AIDA running to no storage again |
|
Fri Mar 12 10:09:19 2021, LS, CA, Friday 12th March 11.00- 17x
|
11.00(Germany) System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment1)
Spectra rates (attachment2)
FEE temps (attachment3)
Leakage currents, written to google sheet (attachment4)
Merger~ 4.9M items/s
Tapeserver ~17MB/s
In MBS control terminal, connection has been closed intentionally since this morning (file S452f160),
AIDA has been taken out of the timesorter due to the high data rate, buffers were full
AIDA cannot be seen in ucesb or Go4.
13.00 System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment5)
Spectra rates (attachment6)
FEE temps (attachment7)
Leakage currents, written to google sheet (attachment8)
Merger~ 5.1M items/s
Tapeserver ~18MB/s
No timestamp related errors this shift
13.37 AIDA MBS control restarted R33_388
System wide checks same as previous time
13.54 beam stopped for access, file R33_396
13.57 seen recent batch of bad timestamp errors in new merger terminal (attachment9) should be around R33_397
Analysed R33_395, 396, 397, 398 no timewarps
14.29 beam back R33_415
14.54 see more bad timestamps in new merger terminal (attachment10)
14.55 System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment11)
Spectra rates (attachment12)
FEE temps (attachment13)
Leakage currents, written to google sheet (attachment14)
Merger~ 5.0M items/s
Tapeserver ~17MB/s
16.05 AIDA included back into timesorter, AIDA scalers now seen in ucesb, file R33_463
16.15 CA takes over until 18:00
17:11 System wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Base Current Difference
aida07 fault 0xcdd6 : 0xcdd7 : 1
White Rabbit error counter test result: Passed 11, Failed 1
17:13 FEE64 temps ok - attachment 15
Statistics ok - attachment 16
bias and leakage currents ok - attachment 17 |
|
Mon May 17 19:05:23 2021, LS, Monday 17th May 20.00-24.00 21x
|
20.30
Checked stats, seem okay (Fig 1)
Ran R14_99 through analyser, dead times all good (Fig 2)
Merger rate 1.8M items/s
Tape server 6.4MB/s
21.00
Checked stats seem okay (Fig 3)
Ran R14_106 through analyser, dead times all good (Fig 4)
21.30
Checked stats seem okay (Fig 5)
Ran R14_112 through analyser, dead times all good (Fig 6)
22.00
Stats okay (Fig 7)
FEE temps okay (Fig 8)
Rate spectra (Fig 9)
Leakage currents (Fig 10)
System wide checks:
Clock status all passed
White rabbit 1 fail:
Base Current Difference
aida05 fault 0x4da : 0x4e7 : 13
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp all passed
Merger rate 2.4M items/s
Tape server 6.3MB/s
Ran R14_119 through analyser, dead times all good (Fig 11)
22.30
Checked stats seem okay (Fig 12)
Ran R14_123 through analyser, dead times all good (Fig 13)
23.00
Checked stats seem okay (Fig 14)
Ran R14_129 through analyser, dead times all good (Fig 15)
23.30
Checked stats seem okay (Fig 16)
Ran R14_135 through analyser, dead times all good (Fig 17)
24.00
Stats okay (Fig 18)
FEE temps okay (Fig 19)
Leakage currents (Fig 20)
System wide checks:
Clock status all passed
White rabbit 1 fail:
Base Current Difference
aida05 fault 0x4da : 0x4eb : 17
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp all passed
Merger rate 2.5M items/s
Tape server 6.5MB/s
Ran R14_142 through analyser, dead times all good (Fig 21) |
|
Tue May 18 18:47:25 2021, LS, Tuesday 18th May 20:00 - 24:00 24x
|
19.50
Stats okay (Fig 1)
FEE temps okay (Fig 2)
Bias okay (Fig 3)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x520 : 32
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~1.8M items/s
Tape server ~7MB/s
Analysed R14_399, deadtimes are okay (Fig 4)
20.30
Stats okay (Fig 5)
Analysed R14_407, deadtimes are okay (Fig 6)
21.00
Stats okay (Fig 7)
Analysed R14_415, deadtimes are okay (Fig 8)
21.30
Stats okay (Fig 9)
Analysed R14_422, deadtimes are okay (Fig 10)
22.00
Stats okay (Fig 11)
FEE temps okay (Fig 12)
Bias okay (Fig 13)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x524 : 36
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~2.3M items/s
Tape server ~8MB/s
Analysed R14_430, deadtimes are okay (Fig 14)
22.30
Stats okay (Fig 15)
Analysed R14_438, deadtimes are okay (Fig 16)
FEE8 deadtime seems to be abit higher (7s previous now 11s, total elapsed time both 232s) but still within 5%
23.00
Stats okay (Fig 17)
Analysed R14_445, deadtimes are okay (Fig 18)
23.30
Stats okay (Fig 19)
Analysed R14_454, deadtimes are okay (Fig 20)
24.00
Stats okay (Fig 21)
FEE temps okay (Fig 22)
Bias okay (Fig 23)
System wide checks okay except:
Base Current Difference
aida05 fault 0x500 : 0x529 : 41
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
The difference has increased over time (not sure if an issue)
Merger rate ~3M items/s
Tape server ~8MB/s
Analysed R14_462, deadtimes are okay (Fig 24) |
|
Wed May 19 18:50:01 2021, LS, Wednesday 19th May 20:00 to 24:00 21x
|
20.00
Stats okay (Fig 1)
Analysed R14_762 deadtimes are okay (Fig 2)
20.30
Stats okay (Fig 3)
Analysed R14_771 deadtimes are okay (Fig 4)
21.00
Stats okay (Fig 5)
FEE temps okay (Fig 6)
Bias (Fig 7)
Rate spectra (Fig 8)
System wide checks okay except white rabbit 1 fail:
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~2.8M items/s
Tape server ~10MB/s
Analysed R14_781 deadtimes are okay (Fig 9)
21.30
Stats okay (Fig 10)
Analysed R14_789 deadtimes are okay (Fig 11)
22.00
Stats okay (Fig 12)
Analysed R14_799 deadtimes are okay (Fig 13)
22.30
Stats okay (Fig 14)
Analysed R14_807 deadtimes are okay (Fig 15)
23.00
Stats okay (Fig 16)
FEE temps okay (Fig 17)
Bias (Fig 18)
System wide checks okay except white rabbit 1 fail:
Base Current Difference
aida05 fault 0x500 : 0x569 : 105
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Merger rate ~3M items/s
Tape server ~10MB/s
Analysed R14_819 deadtimes are okay (Fig 19)
23.30
Stats okay (Fig 20)
Analysed R14_827 deadtimes are okay (Fig 21) |
|
Sat Apr 17 07:28:02 2021, LPG, TD, Saturday 08:00 - 12:00 9x
|
08:30 CEST
HV and leakage currents: elog:233/1
Detector rates: elog:233/2
Temperatures: elog:233/3
WR status:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
FPGA status:
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
09:20 CEST
Stats: ok!
DB: No faults found
ucesb: ok!
09:40 CEST
Beam will be stopped in order to increase intensity. Expected to be around 3-4 hours.
For now, we still get implants when Beam spill is on. Seems to be fluctuating as they play around.
10:00 CEST
Still getting beam, it is fluctuating in intensity
Stats: ok!
DB: No faults found
ucesb: ok!
HV and leakage currents: elog:233/4
Detector rates: elog:233/5
Temperatures: elog:233/6
Clock check: ok!
ADC calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
WR check: ok!
FPGA check:
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory check: ok!
10:40 CEST
Stats: ok!
DB: No faults found
ucesb: ok!
11:00 CEST
Stats: ok!
DB: No faults found
ucesb: ok!
11:30 CEST
Stats: ok!
DB: No faults found
ucesb: ok!
12:00 BST
Stats: ok!
DB: No faults found
ucesb: ok!
HV and leakage currents: elog:233/7
Detector rates: elog:233/8
Temperatures: elog:233/9
Clock check: ok!
ADC calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
WR check:
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bce : 1
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
FPGA check:
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
Memory check: ok! |
|
Tue Apr 20 14:56:38 2021, LPG, Tuesday 20th 16:00-20:00 15x
|
16:00
New shift, ready to go! Beam stopped just as I started... Waiting for update.
Beam back at 16:08, ucesb scalers shown in elog:253/1
Database ok!
Stats ok!
ucesb ok!
Temperatures ok! elog:253/2
Statistics ok! elog:253/3
Leakage currents ok! Steadily rising, day/night effect? - elog:253/4 and elog:253/5
16:21
Online histograms zeroed in MIDAS. Same done in MBS DAQ.
16:45
Database ok!
Stats ok!
ucesb ok!
System wide checks:
Clock ok!
ADC 1 failed:
FEE64 module aida06 failed
WR ok!
FPGA ok!
Memory ok!
17:15
Database ok!
Stats ok!
ucesb ok!
17:45
Database ok!
Stats ok!
ucesb ok!
18:15
Database ok!
Stats ok!
ucesb ok!
System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!
MIDAS Rates ok! elog:253/6
Temperatures ok! elog:253/7
Leakage currents ok, but still slowly rising! Topped out at ~20:30 yesterday elog:253/8
18:42
Message in the Merger terminal window (elog:253/9)
MERGE Actor (15671): Working with 0 from 12 data sources
19:05
Database ok!
Stats ok!
ucesb ok!
19:30
Some MIDAS spectra in elog:253/10 and elog:253/11
Database ok!
Stats ok!
ucesb ok!
19:50
Database ok!
Stats ok!
ucesb ok! elog:253/12
System wide checks:
Clock ok!
ADC 1 fail:
FEE64 module aida06 failed
Calibration test result: Passed 11, Failed 1
WR ok!
FPGA ok!
Memory ok!
MIDAS Rates ok! elog:253/13
Temperatures ok! elog:253/14
Leakage currents ok, now plateaued! elog:253/15
19:55 - Handing over to James  |
|
Thu Apr 22 07:06:37 2021, LPG, Thursday 22nd April: 08:00-12:00 8x
|
08:01 - Taking over from Corrigan...
ucesb ok!
DB ok!
Stats ok!
08:30
ucesb ok!
DB ok!
Stats ok!
09:15
ucesb ok! elog:260/4
DB ok!
Stats ok!
Leakage currents ok! elog:260/1
Temperatures ok! elog:260/2
MIDAS Rates ok! elog:260/3
System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x40 : 64
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!
09:45
ucesb ok!
DB ok!
Stats ok!
10:30
ucesb ok!
DB ok!
Stats ok!
11:00
ucesb ok! elog:260/5
DB ok!
Stats ok!
Leakage currents ok! elog:260/6
Temperatures ok! elog:260/7
MIDAS Rates ok! elog:260/8
System wide checks
Clock ok!
ADC ok!
WR ok!
FPGA 1 fail
Base Current Difference
aida12 fault 0x0 : 0x42 : 66
FPGA Timestamp error counter test result: Passed 11, Failed 1
Memory ok!
11:30
ucesb ok! elog:260/5
DB ok!
Stats ok!
11:55
ucesb ok!
DB ok!
Stats ok! |
|
Wed Apr 21 15:01:14 2021, LJW, MA, Wednesday 21st April 16:00-20:00 13x
|
16:30
All System checks okay, except for:
FPGA Timestamp error:
Base Current Difference
aida12 fault 0x0 : 0x1b : 27
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Temperature Checks: See attachment 1
Statistics Checks: See attachment 2
Bias & Leakage Currents: See attachment 3
18:09
All system checks okay, except for:
FPGA Timestamp error:
Base Current Difference
aida12 fault 0x0 : 0x1d : 29
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Temperature checks: See attachment 4
Statistics Checks: See attachment 5
Bias & Leakage Currents: See attachment 6
19:40
All system checks okay, except for:
FPGA Timestamp error:
Base Current Difference
aida12 fault 0x0 : 0x1d : 29
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Temperature checks: See attachment 7
Statistics Checks: See attachment 8
Bias & Leakage Currents: See attachment 9
*Beam being tuned to another experiment (R3B) - all DSSD implants fluctuating back and forth to 0 Hz for around the last hour or so*
22:30
General check
Rates, Voltages, Temperatures, Ucesb are attached 10,11,12,13
Its been noticed that aida04 temp. is incresing to about 65 (red), still below 70.
the system wide check are all ok except FPGA
Base Current Difference
aida12 fault 0x0 : 0x1d : 29
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
|
|
Mon Apr 19 16:58:29 2021, LJW, Monday 19th April 16:00-20:00 Shift 6x
|
System Checks @ ~18:10 :
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0xd052 : 0xd05a : 8
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x23 : 35
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 6 4 0 1 5 2 3 2 3 6 : 35636
aida02 : 27 3 2 0 2 3 2 3 3 3 6 : 36516
aida03 : 33 9 2 1 1 3 2 3 3 3 6 : 36556
aida04 : 11 6 5 4 2 4 3 3 3 3 6 : 37036
aida05 : 15 8 7 4 3 4 4 2 3 3 6 : 36908
aida06 : 13 9 7 3 3 4 2 3 3 3 6 : 36876
aida07 : 1 3 3 2 2 3 1 3 3 3 6 : 36236
aida08 : 29 5 2 1 1 2 2 3 3 3 6 : 36380
aida09 : 3 8 2 0 2 3 2 2 2 4 6 : 36972
aida10 : 9 5 3 3 1 4 2 4 2 3 6 : 36124
aida11 : 5 7 3 2 2 2 2 4 2 3 6 : 35900
aida12 : 24 6 3 1 2 3 2 2 3 3 6 : 36064
System Checks @ ~19:55
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0xdb9a : 0xdb9d : 3
aida06 fault 0xb74f : 0xb752 : 3
aida07 fault 0xd052 : 0xd05e : 12
aida08 fault 0xe4c1 : 0xe4c4 : 3
aida09 fault 0xaf4b : 0xaf4d : 2
White Rabbit error counter test result: Passed 7, Failed 5
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x27 : 39
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 18 10 2 2 1 5 2 3 2 3 6 : 35768
aida02 : 21 9 2 1 2 2 2 3 3 3 6 : 36444
aida03 : 13 5 6 1 2 2 2 3 3 3 6 : 36444
aida04 : 30 11 4 3 2 4 3 3 3 3 6 : 37104
aida05 : 10 9 8 4 4 4 4 2 3 3 6 : 36976
aida06 : 22 17 10 3 2 3 1 4 3 3 6 : 37088
aida07 : 20 8 4 1 2 4 1 2 3 3 6 : 35952
aida08 : 14 4 3 0 1 2 2 3 3 3 6 : 36296
aida09 : 24 8 4 0 1 3 3 3 3 3 6 : 36768
aida10 : 19 8 4 1 1 4 2 3 3 3 6 : 36652
aida11 : 6 6 1 2 3 1 2 4 2 3 6 : 35800
aida12 : 15 7 1 1 2 4 1 2 3 3 6 : 35876 |
|