 Wed May 19 09:06:58 2021, CA, May 16th 00:00 - 08:00 16x
|
16th May 2021 - 00:00 - 08:00 Shift
Author: CA
00:15 System wide checks;
FEE64 module aida06 global clocks failed, 6
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 14, Failed 2
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x714c : 0x7154 : 8
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x3 : 3
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 10 3 1 2 3 2 3 2 3 7 : 39660
aida02 : 29 5 3 1 1 3 2 3 2 3 7 : 39596
aida03 : 15 7 1 1 2 3 2 3 2 3 7 : 39588
aida04 : 44 26 19 5 2 4 3 3 2 2 7 : 38608
aida05 : 20 10 3 0 1 4 2 2 3 3 7 : 40208
aida06 : 27 6 2 1 2 4 2 4 2 3 7 : 40284
aida07 : 25 5 3 1 2 3 2 3 2 3 7 : 39644
aida08 : 23 8 2 3 1 4 2 3 2 3 7 : 39772
aida09 : 23 10 3 2 3 4 1 3 2 3 7 : 39644
aida10 : 18 10 3 2 3 2 2 2 2 4 7 : 41160
aida11 : 16 7 3 2 1 2 2 3 2 3 7 : 39464
aida12 : 21 4 7 1 3 3 3 3 2 3 7 : 40004
aida13 : 25 6 3 2 1 3 1 4 2 3 7 : 39876
aida14 : 20 8 10 1 1 2 2 2 2 4 7 : 41104
aida15 : 26 6 3 3 0 4 3 2 2 3 7 : 39464
aida16 : 9 8 2 4 1 3 1 4 2 3 7 : 39876
00:30 Stats ok - 210516_0030_stats.png
Temps ok - 210516_0030_temps.png
Bias ok - 210516_0030_bias.png
00:38 analyzer output R7_178 - Deadtime at 7.3% - 210516_R7_178_analysis.txt
02:21 system wide checks as above, except;
Base Current Difference
aida05 fault 0x714c : 0x7159 : 13
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x7 : 7
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
02:28 Stats ok - 210516_0230_stats.png
Temps ok - 210516_0230_temps.png
Bias ok - 210516_0230_bias.png
02:33 analyzer output R7_216 - Deadtime at 6.5% - 210516_R7_216_analysis.txt
04:36 system wide checks same as above, except;
Base Current Difference
aida05 fault 0x714c : 0x715c : 16
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x9 : 9
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
04:37 Stats ok - 210516_0430_stats.png
Temps ok - 210516_0430_temps.png
Bias ok - 210516_0430_bias.png
04:44 analyzer output R7_260 - Deadtime at 6.9% - 210516_R7_260_analysis.txt
06:38 system wide checks same as before, except;
Base Current Difference
aida01 fault 0xf294 : 0xf296 : 2
aida02 fault 0xd8ec : 0xd8ee : 2
aida03 fault 0xf001 : 0xf003 : 2
aida04 fault 0xd992 : 0xd994 : 2
aida05 fault 0x714c : 0x7161 : 21
aida06 fault 0x5a49 : 0x5a4a : 1
aida07 fault 0x5aca : 0x5acb : 1
aida08 fault 0xb92e : 0xb92f : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
06:40 Stats ok - 210516_0640_stats.png
Temps ok - 210516_0640_temps.png
Bias ok - 210516_0640_bias.png
06:45 analyzer output R7_298 - Deadtime at 5.8% - 210516_R7_298_analysis.txt |
|
Wed May 19 09:09:27 2021, MS, JS, OH, May 16 08:00-24:00 21x
|
16th May 08:00 - 12:00 shift
Author: MS
08:00
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0xf294 : 0xf296 : 2
aida02 fault 0xd8ec : 0xd8ee : 2
aida03 fault 0xf001 : 0xf003 : 2
aida04 fault 0xd992 : 0xd994 : 2
aida05 fault 0x714c : 0x7163 : 23
aida06 fault 0x5a49 : 0x5a4a : 1
aida07 fault 0x5aca : 0x5acb : 1
aida08 fault 0xb92e : 0xb92f : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0xa : 10
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 7 2 3 2 3 2 4 2 3 7 : 40228
aida02 : 3 8 3 2 2 2 2 4 2 3 7 : 39996
aida03 : 23 9 6 1 0 3 2 4 2 3 7 : 40100
aida04 : 34 27 17 7 2 4 3 3 2 2 7 : 38608
aida05 : 19 9 5 2 2 2 3 3 2 3 7 : 39844
aida06 : 5 5 2 1 0 4 2 4 2 3 7 : 40060
aida07 : 28 2 10 3 2 4 1 4 2 3 7 : 40192
aida08 : 19 5 5 1 2 5 1 3 2 3 7 : 39652
aida09 : 20 8 3 3 1 3 2 3 2 3 7 : 39648
aida10 : 27 10 0 2 2 2 2 3 1 4 7 : 40572
aida11 : 3 3 2 1 2 4 2 3 2 3 7 : 39652
aida12 : 16 7 11 2 3 4 2 2 2 3 7 : 39464
aida13 : 14 10 4 3 1 5 2 2 2 3 7 : 39400
aida14 : 21 9 10 1 1 2 2 3 1 4 7 : 40604
aida15 : 19 8 2 3 0 4 3 2 2 3 7 : 39436
aida16 : 24 5 8 3 3 4 2 2 2 3 7 : 39464
*** Timestamp elapsed time: 225.065 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.038 0.000
1 9.479 0.000
2 0.195 0.000
3 5.921 0.000
4 0.000 11.742
5 0.036 0.000
6 0.013 0.000
7 0.498 0.000
8 0.436 0.000
9 0.000 107.300
10 2.787 0.000
11 0.905 0.000
12 0.831 0.000
13 0.000 55.939
14 0.080 0.000
15 0.267 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
10:00
FEE64 module aida06 global clocks failed, 6
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 14, Failed 2
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0xf294 : 0xf296 : 2
aida02 fault 0xd8ec : 0xd8ee : 2
aida03 fault 0xf001 : 0xf003 : 2
aida04 fault 0xd992 : 0xd994 : 2
aida05 fault 0x714c : 0x7166 : 26
aida06 fault 0x5a49 : 0x5a4a : 1
aida07 fault 0x5aca : 0x5acb : 1
aida08 fault 0xb92e : 0xb92f : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0xa : 10
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 7 6 2 2 3 2 3 2 3 7 : 39716
aida02 : 8 4 3 2 2 3 2 3 2 3 7 : 39600
aida03 : 23 11 5 1 0 4 2 3 1 4 7 : 40740
aida04 : 42 25 16 7 2 4 4 3 2 2 7 : 38864
aida05 : 24 5 5 1 2 2 3 3 1 4 7 : 40824
aida06 : 11 4 4 1 1 4 2 4 2 3 7 : 40172
aida07 : 21 8 5 1 2 3 2 4 2 3 7 : 40196
aida08 : 15 5 6 1 1 4 2 4 2 3 7 : 40228
aida09 : 15 10 4 1 2 3 2 3 2 3 7 : 39660
aida10 : 23 8 2 2 2 2 2 3 1 4 7 : 40572
aida11 : 6 4 4 2 2 4 2 3 2 3 7 : 39736
aida12 : 18 8 8 1 4 4 3 2 2 3 7 : 39720
aida13 : 23 6 2 3 2 5 2 2 2 3 7 : 39436
aida14 : 29 3 11 1 1 2 2 3 1 4 7 : 40604
aida15 : 29 7 2 2 0 4 3 3 2 3 7 : 39948
aida16 : 2 3 6 2 3 4 2 2 2 3 7 : 39296
*** Timestamp elapsed time: 225.065 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.038 0.000
1 9.479 0.000
2 0.195 0.000
3 5.921 0.000
4 0.000 11.742
5 0.036 0.000
6 0.013 0.000
7 0.498 0.000
8 0.436 0.000
9 0.000 107.300
10 2.787 0.000
11 0.905 0.000
12 0.831 0.000
13 0.000 55.939
14 0.080 0.000
15 0.267 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
12:00-16:00
16th May 12:00 - 16:00 shift
Author: JS
11:57 Taking over from Magda. Running full checks.
usbec ok. Max ~1700 Hz 1MHz on DSSD1, DSSD ~ 75%
Current ok 06.410 uA 006.835 uA
Stats good 1Statistics aidas-gsi(6).png
Temps ok 1Temperature and status scan aidas-gsi(6).png
Analysis ok R7_385. Dead time FEE1 a little hight 6%
PAUSE: 166 RESUME: 166
*** Timestamp elapsed time: 196.305 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.044 0.000
1 12.405 0.000
2 0.807 0.000
3 7.260 0.000
4 0.014 0.000
5 0.458 0.000
6 0.027 0.000
7 0.497 0.000
8 0.723 0.000
9 0.000 88.857
10 6.189 0.000
11 1.443 0.000
12 0.474 0.000
13 0.000 35.565
14 0.000 0.000
15 0.147 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1
Base Current Difference
aida01 fault 0xf294 : 0xf296 : 2
aida02 fault 0xd8ec : 0xd8ee : 2
aida03 fault 0xf001 : 0xf003 : 2
aida04 fault 0xd992 : 0xd994 : 2
aida05 fault 0x714c : 0x716e : 34
aida06 fault 0x5a49 : 0x5a4a : 1
aida07 fault 0x5aca : 0x5acb : 1
aida08 fault 0xb92e : 0xb92f : 1
White Rabbit error counter test result: Passed 8, Failed 8
Base Current Difference
aida05 fault 0x0 : 0xa : 10
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 7 4 2 2 3 2 3 1 4 7 : 40712
aida02 : 25 10 7 1 1 4 1 3 2 3 7 : 39556
aida03 : 22 10 5 1 0 4 2 4 2 3 7 : 40216
aida04 : 40 24 17 7 2 5 3 3 2 2 7 : 38736
aida05 : 4 8 3 0 1 3 3 3 1 4 7 : 40768
aida06 : 21 6 6 2 2 3 2 4 2 3 7 : 40228
aida07 : 25 4 6 3 2 3 2 4 2 3 7 : 40260
aida08 : 19 9 4 1 1 4 2 4 2 3 7 : 40244
aida09 : 16 9 4 2 2 3 2 3 2 3 7 : 39688
aida10 : 19 10 5 1 2 2 2 4 1 4 7 : 41100
aida11 : 21 10 2 1 2 3 3 3 2 3 7 : 39908
aida12 : 25 7 7 2 2 3 2 3 2 3 7 : 39756
aida13 : 13 7 3 4 2 5 2 3 2 3 7 : 39964
aida14 : 21 7 9 2 1 2 2 4 1 4 7 : 41116
aida15 : 23 6 2 3 0 4 3 3 2 3 7 : 39948
aida16 : 9 11 10 2 3 4 2 2 2 3 7 : 39452
Tom says Aida09 clock fail is ok as its status bit is "6".
The large white difference for FEE5 is known and has been determined to be ok, a post run investigation will be undertaken.
12:35 -
usbec ok.
Current ok
Stats good
Temps ok
Analysis ok R7_395. Dead time FEE1 still high 6.6%
13:33 -
usbec ok. Max implants ~ 1.8kHz
Current ok 006.850 uA 007.250 uA
Stats - Aida11 runing low < 5k was ~20k overnight
Temps ok
Analysis R7_415. Dead time FEE1 & FEE10 high 10%
14:00
usbec ok - ucesb1.png
Current ok - bias1.png
Stast - Aida11 low -
14:20 aida fee rebooted itself. A powercycle was performed. Upon reboot we are seeing extremely large amounts of noise in the FEEs. Looking at the waveforms we have very large 100kHz pick up in the FEEs. This has resulted in 50% deadtime in many FEEs including the p+n.
15:28 Because of extremely high rates across all FEEs have decided to do a powercycle. Before restarting the FEEs will give them a couple of minutes to cool.
18:00 Since the start of the shift we have been trying to recover the system froma large increase in noise following the crash at ~14:00.
During this time NH has entered the area and inspected the system and also grounded the AIDA snout. This provided us with some improvement on the noise.
The rates are still slightly above where we were before the crash but now appear stable. To counteract the dead time in the n+n strips we have raised the threshold to 0x64 for ASIC4 in all FEEs.
We are now running with around 10% deadtime on FEE4 and less elsewhere for n+n. For p+n most have zero dead time apart from FEE11 which is still noisy.
During the time we were trying to recover the system screenshots were taken of the waveforms. He it could be seen that the 100kHz noise was very apparent. Particularly in the n+n strips.
18:08 System wide checks all ok - bar some ADC but waveforms disabled
Statistics ok - 210516_1809_Stats
Temperatures ok - 210516_1809_Temp
Bias and leakage ok - 210516_1810_Bias
18:37 Performed an ASIC check and now the rates have dropped in all n+n strips. Currently very small amounts of dead time
18:40 Realised this was because it raised the threshold of all strips to 0x64 on the n+n side.
19:25 Removed S452 from 1e2.... drive. Before removing checked with Nic backed up to Lustre. Also verified four ourselves.
Now have around 4.2TB left which will provide around 80 hours of writing
20:16 We noticed iptraf was using around 30% CPU usage. We investigated whether it had any effect on the dead time but from what we have seen it has not.
20:57 System wide checks. Clock ok
Base Current Difference
aida05 fault 0x1552 : 0x1556 : 4
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Statistics ok - 210516_2056_Stats
Temp ok - 210516_2058_Temp
Bias and leakage current ok - 210516_2058_Bias
23:16 System wide checks:
Clock still ok
Base Current Difference
aida05 fault 0x1552 : 0x155a : 8
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Statistics - 210516_2315_Stats
Temperature - 210516_2316_Temp
Bias and leakage current ok - 210516_231 |
|
Thu May 13 07:10:17 2021, CA, TD, May 13th 08:00 - 16:00 shift 34x
|
08:10 all system wide checks ok *except*
all FEE64 modules fail ADC calibration
WR decoder status:
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9db9 : 1
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
08:16 attempt to recalibrate ADCs using FADC align and control - unsuccessful on all
08:18 FEE64 temps ok - attachment 1
bias/leakage currents ok - attachment 2
statistics - attachments 3-8
10:13 all system wide checks ok *except*
all FEE64 modules fail ADC calibration
WR decoder status:
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9db9 : 1
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
10:16 FEE64 temps ok - attachment 9
bias/leakage currents ok - attachment 10
statistics - attachments 11-16
12:16 all system wide checks ok *except*
all FEE64 modules fail ADC calibration
WR decoder status:
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dba : 2
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
12:16 FEE64 temps ok - attachment 17
bias/leakage currents ok - attachment 18
statistics - attachments 19-25
14:08 all system wide checks ok *except*
all FEE64 modules fail ADC calibration
WR decoder status:
Base Current Difference
aida01 fault 0xf932 : 0xf933 : 1
aida02 fault 0x62ec : 0x62ed : 1
aida03 fault 0x8679 : 0x867a : 1
aida04 fault 0xf0e4 : 0xf0e5 : 1
aida05 fault 0x9db8 : 0x9dba : 2
aida06 fault 0x7f18 : 0x7f19 : 1
aida07 fault 0xdd2c : 0xdd2d : 1
aida08 fault 0x1557 : 0x1558 : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FEE64 temps ok - attachment 16
bias/leakage currents ok - attachment 27
statistics - attachments 28-35 |
|
Tue Mar 9 06:58:14 2021, CA, LS, March 9th 08:00-11:00 shift 23x
|
08:00 DAQ continues ok - writing to file R13_120
Merger ok - 4.6M data items/s
TapeServer ok - 43 MB/s
08:03 - all system wide checks ok
- Temperatures ok - attachment 1
- good event statistics ok - attachment 2
- detector bias/leakage currents ok - attachment 3
08:08 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
08:32 CA briefly lost network connection - back now
08:42 DESPEC on file 102
09:24 Rate spectra - attachment 4
09:32 HEC spectra - attachment 5/6
beam off - writing to file R13_243
09:42 merger bad timestamp errors - attachment 7
all system wide checks ok
10:30 NH - Beam was down, I try the new merger from Vic again (R14)
I made a mistake and had to powercycle fees as they got stuck! aida10 seemed to reboot itslef
After fixing my mistakes, all running with new merger and rate 15 MB/s (R14)
It all looked OK even going to MBS/Go4
I reverted back to original hot merger, R15 and look at the data.
Beam is now back and I return AIDA to shifters :)
11.30 System wide checks okay except ADC calibration:
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Reran calibration for both FEEs but still fail
Statistics (attachment8)
Rate spectra (attachment9)
FEE temps (attachment10)
Leakage currents written to sheet (attachment11)
Tapeserver ~33MB/s
Merger ~ 3.4M items/s
AIDA file R15_92
11.55 Beam off for a couple minutes but back now
12.00 New Merge terminal disappeared so stopped daq and restarted, and also restarted MBS back working
Now writing to R16
Merger ~ 4.5M items/s
Tapeserver ~44MB/s
12.41 no beam
13.00 DAQ crashed (aida03 no responding), AIDA recovered from power cycle while beam is down
Beam is back now, forwarding to MBS
Merger ~ 45M items/s
Tapeserver ~44MB/s
Writing to R17
13.20 System wide checks okay except ADC calibration:
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment12)
Rate spectra (attachment13)
FEE temps (attachment14)
Leakage currents written to sheet (attachment15)
Tapeserver ~45MB/s
Merger ~ 4.4M items/s
14.27 beam off
file R17_115
back a couple minutes later
14.40 no beam again back at 14.46
15.40 System wide checks okay except ADC calibration:
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment16)
Rate spectra (attachment17)
FEE temps (attachment18)
Leakage currents written to sheet (attachment19)
Tapeserver ~44MB/s
Merger ~ 4.4M items/s
15.52 beam down R17_221, back at 15.58 R17_229
16.07 Beam down
16:20 Removed S480 files from /media/1e... and /media/SecondDrive
Set merger to no output and stopped tapedata.
Changed symbolic link of /TapeData to /media/1e.....
Restarted tapedata and the merger with outputting to file R20
Local DESPEC was made aware of this before carrying out and stopped the run.
Beam back
17.12 beam down for a couple of minutes R20_90
17.40 System wide checks okay except ADC calibration:
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment20)
Rate spectra (attachment21)
FEE temps (attachment22)
Leakage currents written to sheet (attachment23)
Tapeserver ~45MB/s
Merger ~ 4.5M items/s
|
|
Sun Mar 7 07:23:26 2021, CA, LS, March 7th 14x
|
08:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xa
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
08:30 - all system wide checks ok
- Temperatures ok - attachment 1
- good event statistics ok - attachment 2
- detector bias/leakage currents ok - attachment 3
DAQ continues to run - R1_1148
Merger ok - 4.6M data items/s
TapeServer ok - 43MB/s
08:55 Giovanna Benzoni - confirms contribution of light ions - attachment 4 see AOQ plots
09:19 zeroed all histograms
writing to file R1_1204
FRS on file 72
DESPEC to add AIDA XY hit patterns to online analysis histograms
09:35 beam stop
09:56 beam back
writing to file R1_1249
rate spectra - attachment 5
09:30 AIDA DAQ crashed - lost connection to aida05
power cycle and reset
09:40 AIDA ok now, writing to file R2
09:50 LS takes over
11.10 (German time)
System wide checks produces some fails:
FEE64 module aida06 global clocks failed, 6 (screenshot6)
This error is seen in RIKEN too, tried a RESYNC from master timestamp tab but error still there
FEE64 module aida06 failed
FEE64 module aida07 failed
Calibration test result: Passed 10, Failed 2 (screenshot7)
Tried a recalibration for the two FEEs in FADC align & control tab but error still there
Rest of the checks pass
Statistics (screenshot8)
FEE temps okay (screenshot9)
Leakage currents written to sheets (screenshot10)
Merger ~4.5M data items/s
TapeServer ~ 45MB/s
13.00 System wide checks produces same fails as last time:
FEE64 module aida06 global clocks failed, 6
FEE64 module aida06 failed
FEE64 module aida07 failed
Calibration test result: Passed 10, Failed 2
Rest of the checks are okay
Statistics (screenshot11)
Spectra rate (screenshot12)
FEE temps okay (screenshot13)
Leakage currents written to sheets (screenshot14)
Merger ~4.5M data items/s
TapeServer ~ 45MB/s
There is 1.5TB left on current drive
14.00 beam stopped to look at bPlast, have use the time to change slow comparator threshold
Slow comparators in FEE9-12 raised to 0xd
|
|
Fri Mar 5 23:21:59 2021, CA, OH, March 6th 2021 00:00 - 08:00 shift 18x
|
00:23 206Hg setting - No beam in AIDA yet
AIDA not writing to file
Data being forwarded to MBS
Fragments expected tonight (?)
00:45 GSI DAQ restart
FEE64 Temps ok - attachment 1
All system wide checks ok
01:00 Good event statistics - attachment 2
01:48 detector biases/ leakage currents - ok - attachment 3
02:10 beam on, plug removed
beam delivered to AIDA
rates spectra - attachment 4
peaks at ~ 10 counts/channel in DSSD1 c.2000Hz
02:20 beam stop
02:26 beam restart - rate spectra - attachment 5
02:43 high energy channel spectra - attachment 6
saved the high energy spectra to layout 2 in spectrum browser
03:00 beam stop
03:08 temperatures ok - attachment 7
statistics ok - attachment 8
bias/leakage currents ok - attachment 9
NewMerger ok - ~4.5M data items merge rate
Tape service ok - ~46MB/s, NoStorage
Data forwarding to MBS ok
all system wide checks ok
04:24 beam is back
rate spectra - attachment 10
04:51 206Hg implanting in DSSD2
Rate spectra - attachment 11
HEC spectra - attachment 12
05:08 temperatures ok - attachment 13
statistics ok - attachment 14
detector bias/leakage currents ok - attachment 15
All system wide checks ok
NewMerger ok - ~4.5M data items merge rate
Tape service ok - ~46MB/s, NoStorage
Data forwarding to MBS ok
05:45 beam stopped
05:50 beam back
06:45 beam running, but no implants in AIDA
possible issues with beam delivery to S4
NewMerger ok - ~4.5M data items merge rate
Tape service ok - ~46MB/s, NoStorage
Data forwarding to MBS ok
07:10 temperatures ok - attachment 16
statistics ok - attachment 17
detector bias/leakage currents ok - attachment 18
all system wide checks OK
|
|
Sat Mar 13 07:08:52 2021, CA, TD, LS, March 13th 08:00 - 17:00 20x
|
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
08:09 System wide checks ok *except*
aida09 fails clock check
aida09 calibration failed
08:14 FEE64 temperatures ok - attachment 1
good event statistics ok - attachment 2
detector bias/leakage current ok - attachment 3
08:22 Merger 4.3M items/s
Tapeserver 14 MB/s
08:24 DAQ ok, writing to file R43_45
08:39 rate spectra - attachment 4
aida09 spectrum not showing, however continues to collect statistics ok
10:03 System wide checks ok *except*
aida09 fails clock check
aida09 calibration failed
FEE64 temperatures ok - attachment 5
good event statistics ok - attachment 6
detector bias/leakage current ok - attachment 7
10:10 Merger 4.5M items/s
Tapeserver 14 MB/s
data forwarding to MBS ok
10:11 writing to file R43_91
12.00 (LS)
System wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Base Current Difference
aida07 fault 0x829e : 0x829f : 1
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
no FGPA timestamp errors all passed
Statistics (attachment8)
Spectra rate (attachment9), aida09 still does not show histogram, but on run control are enabled for all fees. still
showing good events and other information (temperatures etc)
FEE temps (attachment10)
Leakage currents written to sheets (attachment11)
Merger~4.6M items/s
Tapeserver~14MB/s
12.36 bunch of bad timestamp errors in the new merger terminal (attachment 12) around file R43_152
this error is still occurring at 12.45 will continue to monitor, all other system checks are reporting as normal
13.50 beam down file R43_183
beam back a couple of minutes later
14.00 System wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
no white rabbit or FPGA errors all passed
Statistics (attachment13)
Spectra rate (attachment14)
FEE temps (attachment15)
Leakage currents written to sheets (attachment16)
Merger~4.6M items/s
Tapeserver~14MB/s
Writing to MBS okay, still seeing bad timestamp errors in the new merger terminal continuing to monitor and update
statistics tab every 20 to 30 minutes
14.12 Analysed files R43_185,186,187 from timestamp error in ucesb, but see no timewarps
14.30 have not seen any bad timestamp errors in new merger terminal for a while, unsure if something has been changed which
has stopped them
15.25 beam down
15.53 beam back
16.00 moved back to writing to /media/SecondDrive which should last until the end of the experiment
now writing R46
16.10 System wide checks okay except:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment17)
Spectra rate (attachment18)
FEE temps (attachment19)
Leakage currents written to sheets (attachment20)
Merger~4.7M items/s
Tapeserver~14MB/s
Writing to MBS okay |
|
Thu Mar 11 23:07:23 2021, CA, March 12th 00:00 - 08:00 shift 15x
|
00:10 DAQ continues OK - file R30_372
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:17 System wide checks all OK
FEE64 Temperatures ok - attachment 1
Good event statistics ok - attachment 2
detector bias and leakage currents ok - attachment 3
00:20 DESPEC on run 145
00:24 Merger ok ~45M items/s
Tapeserver ok ~15MB/s
01:00 aida07 crashed
recovered, but all FEE64 crash shortly after
called OH - stop DAQ, TapeService, Merger
01:20 all FEE64 powercycled, AIDA restart
01:30 AIDA recovered, DAQ now running - writing to R33
system wide checks ok *except*
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
01:52 FEE64 Temperatures ok - attachment 4
-had to reload a few times to work, but otherwise ok
Good event statistics - attachment 5
-aida05 and aida08 running faster than before
detector bias and leakage currents ok - attachment 6
02:04 writing to file R33_34
Data forwarding to MBS ok
AIDA ASIC settings ok
02:11 beam off
02:13 attempted to recalibrate aida07 and aida09 in FADC Align and Control - calibration still fails
02:32 beam back - writing to file R33_48
04:30 DESPEC having issues with Go4 crashing
ucesb reports AIDA/FATIMA/bplast timewarp events
System wide checks:
Clock error:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Calibration:
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
White Rabbit:
Base Current Difference
aida07 fault 0xcdd1 : 0xcdd2 : 1
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FEE64 Temperatures ok - attachment 7
Good event statistics ok - attachment 8
detector bias and leakage currents ok - attachment 9
04:52 analyzer output for R33_113 - attachment 10
no timewarps
aida06 dead time? (ignore idle time and rates)
05:02 Merger 5M data items/s
TapeServer 17 MB/s
05:15 DESPEC believe issue with bplast TAMEX causing ucesb issues/Go4 crash
They disable bplast and FATIMA TAMEX histograms in their online analysis - ucesb/go4 much more stable now
05:25 rates spectra - attachment 11
05:29 error message in MBS relay terminal - otherwise data forwarding to MBS ok - attachment 12
06:29 system wide checks:
Clock error:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Calibration:
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
White Rabbit:
Base Current Difference
aida05 fault 0x7a56 : 0x7a58 : 2
aida06 fault 0x65bb : 0x65bd : 2
aida07 fault 0xcdd1 : 0xcdd4 : 3
aida08 fault 0x2ab3 : 0x2ab5 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
rest ok
06:32 FEE64 Temperatures ok - attachment 13
Good event statistics ok - attachment 14
detector bias and leakage currents ok - attachment 15
06:41 Merger 5.1M data items/s
TapeServer 17 MB/s
Data forwarding to MBS ok
08:54 restart MBS relay, requested by NH
|
|
Tue Mar 9 23:12:17 2021, CA, March 10th 00:00 - 08:00 20x
|
00:13 beam back, but some spills 'missing'
DESPEC keeping beam as is, but may stop later if it worsens
00:15 DAQ continues OK - file R20_622
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:20 System wide checks all OK *except*
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
00:24 still no bad timestamp errors in NewMerger since 16:30 UTC
00:30 FEE64 Temperatures OK - attachment 1
Good event statistics OK - attachment 2
Detector bias & leakage currents OK - attachment 3
Merger OK - 4.2M data items/s
TapeServer OK - 45 Mb/s
01:30 rate spectra - attachment 4
HEC spectra - attachment 5 & 6
note aida04 spectrum still not showing
02:07 beam off - file R20_760
02:11 beam back - file R20_764
02:44 System wide checks all OK *except*
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
still no bad timestamp errors in NewMerger since 16:30 UTC
FEE64 Temperatures OK - attachment 7
Good event statistics OK - attachment 8
Detector bias & leakage currents OK - attachment 9
Merger OK - 4.2M data items/s
TapeServer OK - 44 Mb/s
04:16 bad timestamp errors in NewMerger terminal, first since 16:30 UTC - attachment 10
System wide checks all OK *except*
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
FEE64 Temperatures OK - attachment 11
Good event statistics OK - attachment 12
Detector bias & leakage currents OK - attachment 13
05:03 Merger OK - 4.5M data items/s
TapeServer OK - 43 Mb/s
no bad timestamp errors for ~40 mins, data forwarding to MBS ok at usual rate
06:11 rates spectra - attachment 14
06:14 System wide checks
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
Check WR decoder status
Base Current Difference
aida05 fault 0x1591 : 0x1593 : 2
aida06 fault 0xe23c : 0xe23e : 2
aida07 fault 0x743 : 0x745 : 2
aida08 fault 0xf84e : 0xf850 : 2
aida09 fault 0x5449 : 0x544a : 1
White Rabbit error counter test result: Passed 7, Failed 5
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
still no further bad timestamp errors in NewMerger terminal
collected all WR and FPGA errors from baseline, system wide checks ok
FEE64 Temperatures OK - attachment 15
Good event statistics OK - attachment 16
Detector bias & leakage currents OK - attachment 17
06:24 AIDA writing to file R20_1080
Merger OK - 4.5M data items/s
TapeServer OK - 46 Mb/s
06:26 another burst of bad timestamp errors - attachment 18
performed system wide checks again - all ok aside from aida07/aida10 calibration
07:32 HEC spectra - attachments 19/20
07:35 no beam
|
|
Tue Apr 5 12:46:52 2022, TD, MSL type BB18(DS)-1000 triple - cable lengths
|
OH/NH disassembling current AIDA triple snout and DSSSD stack.
Length of ribbon cables (including connector) outside snout as follows
p+n junction side 9cm & 15cm
n+n ohmic side 10.5cm & 16cm
Update
p+n junction side: 6.5cm, 10.5 cm (used 7.5, 11.5 for 1cm slack)
n+n ohmic side: 11.0 cm, 15.0 cm (used 12.0, 16.0 for 1cm slack)
Measurement from SAMTEC connector to exit from the rubber seals in AIDA snout. |
|
Thu Mar 25 18:04:51 2021, TD, MSL type BB18(DS) 24cm x 8cm cabling & bias configuration
|
|
|
Thu Oct 12 14:01:43 2023, TD, MSL type BB18 24cm x 8cm DSSSD test - update
|
DSSSD MSL type BB18 24cm x 8cm 3208-3/3208-21/3208-22
FEE64 configuration see https://elog.ph.ed.ac.uk/AIDA/872 attachment 2
Bias -150V -6.590uA ambient temperature +24.7 deg C d.p. +13.7 deg C RH 50.3%
BNC PB-5
amplitude 10.0V
attenuation x10
decay time 1ms
tail pulse
frequency 25Hz
PB-5 output direct to p+n junction side FEE64 aida01 or aida12, or n+n Ohmic side FEE64 aida02
aida01 1.8.L pulser peak width 61 ch FWHM ~ 46keV FWHM => 5s threshold 98keV
aida12 1.8.L pulser peak width 56 ch FWHM ~ 42keV FWHM => 5s threshold 89keV
aida02 1.8.L pulser peak width 102 ch FWHM ~ 77keV FWHM => 5s threshold 163keV
slow comparator 0xa 100keV ( all p+n junction FEE64s )
0xf 150keV ( all n+n Ohmic FEE64s )
per FEE64 Rate spectra - attachment 1
p+n FEE64s ( aida010, aida01, aida09, aida12, aida03, aida11 ) rates dominated by hot channels, other channels typically <1Hz ( 25Hz pulser to aida12 )
n+n FEE64s ( aida02, aida04 ) rates ~ 10-20Hz/channel
Note aida06 and aida08 are not connected to anything and should be ignored
ADC data item stats - attachment 2
For further information see https://elog.ph.ed.ac.uk/AIDA/906 and https://elog.ph.ed.ac.uk/AIDA/907
To Do
- repair/replace Honeywell HSS-DPS dew point sensor
USB-controlled ac mains relay interlock currently overrriden
do NOT operate AIDA unattended
- aida04 asic #1 u/s - replace ASIC mezzanine
- electrically isolated test signal distribution box req'd
- aida10 asic #4
v. high rates observed and large signal transients
cause unclear ... ASIC/adaptor PCB/cabling/Si wafer ?
- extended background alpha run to check all DSSSD bond wires
pulser OFF
slow comparator 0x64
- bPlas + 2x triple DSSSD + bPlas stack assembly and test
- all up, in beam test |
|
Thu Oct 21 09:08:41 2021, OH, MIDAS start up messages
|
| |
|
Thu Aug 19 10:47:13 2021, TD, MIDAS spectra
|
|
|
Thu Dec 19 13:09:32 2019, TD, MIDAS app launcher command lines
|
App Path
TclHttpd @ 8015 for AIDA /MIDAS/Linux/startup/HTTPD
TclHttpd @ 8115 for AIDA /MIDAS/Linux/startup/HTTPD@8115
TapeServer /MIDAS/Linux/startup/TapeServer
New Merger for AIDA /MIDAS/Linux/startup/NewMerger
MBS Relay /MIDAS/Linux/startup/datarelaymbs |
|
Tue Jun 28 10:11:35 2022, OH, NH, MIDAS Data Aq V10
|
11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics
Patrick has produced V10 which removes these.
When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
WR items for 0xE - attachment 1
WR items for 0x7 - attachment 2
However we see in the NewMerger terminal the message shown in attachment 3 frequently.
Also we note that the merger time error counter is also going up.
Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
Are we having dead time issues which is causing time warps?
Does each buffer from the MIDAS Data Acq start with a full WR timestamp?
aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way? |
|
Wed Mar 3 15:02:19 2021, CA, MBS command line startup
|
/MIDAS/Linux/startup/datarelaymbs |
|
Tue Apr 2 09:42:20 2019, NH, MBS Server setup
|
MBS is currently configured and accepting data:
x86l-94 : AIDA Foreign data receiver
x86l-4 : DESPEC Time sorter
The time sorter is complaining about AIDA time warps (around 1 Hz) it is assumed this is related to other merger / timestamping issues being worked on by Vic |
|
Mon Apr 1 10:20:20 2019, CA, NH, TD, MBS Relay startup
|
To start the MBS Data Relay, click on the top rightmost of the MIDAS icons (MBS relay, Data Relay => MBS) in
Desktop 3. (attachment 1)
This will bring up the Data Relay MBS terminal.
Ensure the path defined in the datarelaymbs is set to the correct directory (shown in attachment 2) |
|
Thu Jan 24 17:16:32 2019, CA, TD, NH, VP, MACB time switch settings
|
AIDA@DESPEC MACB time switch settings before and after WR are as follows.
Attachment 1 - MACB 1-4 from left to right
MACB number: 1 2 3 4
100MHZ clock setting (before WR): 0 2 2 2
White Rabbit setting (after WR): 3 3 3 3 |
|