| ID |
Date |
Author |
Subject |
|
459
|
Mon May 16 07:03:16 2022 |
MS, OH | Monday 16th May 08:00-16:00 |
08:00 Took over the night shift from Tom (Attachments 1-3)
10:00: Attachement 4-6
around 10:30: Beam went off (Attachment 7)
11:00: Beam is back
12:00 Attachments 8-10
14:00 Attachements 11-13
15:50 Attachments 14-17 |
| Attachment 1: Screenshot_2022-05-16_at_07-59-51_Statistics_aidas-gsi.png
|
|
| Attachment 2: Screenshot_2022-05-16_at_08-00-15_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 3: Screenshot_from_2022-05-16_08-01-20.png
|
|
| Attachment 4: Screenshot_2022-05-16_at_09-59-15_Statistics_aidas-gsi.png
|
|
| Attachment 5: Screenshot_2022-05-16_at_09-59-45_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 6: Screenshot_from_2022-05-16_10-01-47.png
|
|
| Attachment 7: Screenshot_2022-05-16_at_10-47-01_ucesb.png
|
|
| Attachment 8: Screenshot_2022-05-16_at_11-59-53_Statistics_aidas-gsi.png
|
|
| Attachment 9: Screenshot_2022-05-16_at_12-00-19_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 10: Screenshot_from_2022-05-16_12-00-57.png
|
|
| Attachment 11: Screenshot_2022-05-16_at_14-00-56_Statistics_aidas-gsi.png
|
|
| Attachment 12: Screenshot_2022-05-16_at_14-01-19_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 13: Screenshot_from_2022-05-16_14-01-56.png
|
|
| Attachment 14: Screenshot_2022-05-16_at_15-47-58_Statistics_aidas-gsi.png
|
|
| Attachment 15: Screenshot_2022-05-16_at_15-48-15_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 16: Screenshot_2022-05-16_at_15-48-34_Lost_activity_monitor_aidas-gsi.png
|
|
| Attachment 17: Screenshot_from_2022-05-16_15-49-34.png
|
|
|
329
|
Wed May 19 09:09:27 2021 |
MS, JS, OH | May 16 08:00-24:00 |
16th May 08:00 - 12:00 shift
Author: MS
08:00
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0xf294 : 0xf296 : 2
aida02 fault 0xd8ec : 0xd8ee : 2
aida03 fault 0xf001 : 0xf003 : 2
aida04 fault 0xd992 : 0xd994 : 2
aida05 fault 0x714c : 0x7163 : 23
aida06 fault 0x5a49 : 0x5a4a : 1
aida07 fault 0x5aca : 0x5acb : 1
aida08 fault 0xb92e : 0xb92f : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0xa : 10
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 7 2 3 2 3 2 4 2 3 7 : 40228
aida02 : 3 8 3 2 2 2 2 4 2 3 7 : 39996
aida03 : 23 9 6 1 0 3 2 4 2 3 7 : 40100
aida04 : 34 27 17 7 2 4 3 3 2 2 7 : 38608
aida05 : 19 9 5 2 2 2 3 3 2 3 7 : 39844
aida06 : 5 5 2 1 0 4 2 4 2 3 7 : 40060
aida07 : 28 2 10 3 2 4 1 4 2 3 7 : 40192
aida08 : 19 5 5 1 2 5 1 3 2 3 7 : 39652
aida09 : 20 8 3 3 1 3 2 3 2 3 7 : 39648
aida10 : 27 10 0 2 2 2 2 3 1 4 7 : 40572
aida11 : 3 3 2 1 2 4 2 3 2 3 7 : 39652
aida12 : 16 7 11 2 3 4 2 2 2 3 7 : 39464
aida13 : 14 10 4 3 1 5 2 2 2 3 7 : 39400
aida14 : 21 9 10 1 1 2 2 3 1 4 7 : 40604
aida15 : 19 8 2 3 0 4 3 2 2 3 7 : 39436
aida16 : 24 5 8 3 3 4 2 2 2 3 7 : 39464
*** Timestamp elapsed time: 225.065 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.038 0.000
1 9.479 0.000
2 0.195 0.000
3 5.921 0.000
4 0.000 11.742
5 0.036 0.000
6 0.013 0.000
7 0.498 0.000
8 0.436 0.000
9 0.000 107.300
10 2.787 0.000
11 0.905 0.000
12 0.831 0.000
13 0.000 55.939
14 0.080 0.000
15 0.267 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
10:00
FEE64 module aida06 global clocks failed, 6
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 14, Failed 2
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0xf294 : 0xf296 : 2
aida02 fault 0xd8ec : 0xd8ee : 2
aida03 fault 0xf001 : 0xf003 : 2
aida04 fault 0xd992 : 0xd994 : 2
aida05 fault 0x714c : 0x7166 : 26
aida06 fault 0x5a49 : 0x5a4a : 1
aida07 fault 0x5aca : 0x5acb : 1
aida08 fault 0xb92e : 0xb92f : 1
White Rabbit error counter test result: Passed 8, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0xa : 10
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 7 6 2 2 3 2 3 2 3 7 : 39716
aida02 : 8 4 3 2 2 3 2 3 2 3 7 : 39600
aida03 : 23 11 5 1 0 4 2 3 1 4 7 : 40740
aida04 : 42 25 16 7 2 4 4 3 2 2 7 : 38864
aida05 : 24 5 5 1 2 2 3 3 1 4 7 : 40824
aida06 : 11 4 4 1 1 4 2 4 2 3 7 : 40172
aida07 : 21 8 5 1 2 3 2 4 2 3 7 : 40196
aida08 : 15 5 6 1 1 4 2 4 2 3 7 : 40228
aida09 : 15 10 4 1 2 3 2 3 2 3 7 : 39660
aida10 : 23 8 2 2 2 2 2 3 1 4 7 : 40572
aida11 : 6 4 4 2 2 4 2 3 2 3 7 : 39736
aida12 : 18 8 8 1 4 4 3 2 2 3 7 : 39720
aida13 : 23 6 2 3 2 5 2 2 2 3 7 : 39436
aida14 : 29 3 11 1 1 2 2 3 1 4 7 : 40604
aida15 : 29 7 2 2 0 4 3 3 2 3 7 : 39948
aida16 : 2 3 6 2 3 4 2 2 2 3 7 : 39296
*** Timestamp elapsed time: 225.065 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.038 0.000
1 9.479 0.000
2 0.195 0.000
3 5.921 0.000
4 0.000 11.742
5 0.036 0.000
6 0.013 0.000
7 0.498 0.000
8 0.436 0.000
9 0.000 107.300
10 2.787 0.000
11 0.905 0.000
12 0.831 0.000
13 0.000 55.939
14 0.080 0.000
15 0.267 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
12:00-16:00
16th May 12:00 - 16:00 shift
Author: JS
11:57 Taking over from Magda. Running full checks.
usbec ok. Max ~1700 Hz 1MHz on DSSD1, DSSD ~ 75%
Current ok 06.410 uA 006.835 uA
Stats good 1Statistics aidas-gsi(6).png
Temps ok 1Temperature and status scan aidas-gsi(6).png
Analysis ok R7_385. Dead time FEE1 a little hight 6%
PAUSE: 166 RESUME: 166
*** Timestamp elapsed time: 196.305 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.044 0.000
1 12.405 0.000
2 0.807 0.000
3 7.260 0.000
4 0.014 0.000
5 0.458 0.000
6 0.027 0.000
7 0.497 0.000
8 0.723 0.000
9 0.000 88.857
10 6.189 0.000
11 1.443 0.000
12 0.474 0.000
13 0.000 35.565
14 0.000 0.000
15 0.147 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 15, Failed 1
FEE64 module aida09 failed
Calibration test result: Passed 15, Failed 1
Base Current Difference
aida01 fault 0xf294 : 0xf296 : 2
aida02 fault 0xd8ec : 0xd8ee : 2
aida03 fault 0xf001 : 0xf003 : 2
aida04 fault 0xd992 : 0xd994 : 2
aida05 fault 0x714c : 0x716e : 34
aida06 fault 0x5a49 : 0x5a4a : 1
aida07 fault 0x5aca : 0x5acb : 1
aida08 fault 0xb92e : 0xb92f : 1
White Rabbit error counter test result: Passed 8, Failed 8
Base Current Difference
aida05 fault 0x0 : 0xa : 10
aida12 fault 0x0 : 0x3 : 3
aida13 fault 0x0 : 0x4d : 77
FPGA Timestamp error counter test result: Passed 13, Failed 3
Returned 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 7 4 2 2 3 2 3 1 4 7 : 40712
aida02 : 25 10 7 1 1 4 1 3 2 3 7 : 39556
aida03 : 22 10 5 1 0 4 2 4 2 3 7 : 40216
aida04 : 40 24 17 7 2 5 3 3 2 2 7 : 38736
aida05 : 4 8 3 0 1 3 3 3 1 4 7 : 40768
aida06 : 21 6 6 2 2 3 2 4 2 3 7 : 40228
aida07 : 25 4 6 3 2 3 2 4 2 3 7 : 40260
aida08 : 19 9 4 1 1 4 2 4 2 3 7 : 40244
aida09 : 16 9 4 2 2 3 2 3 2 3 7 : 39688
aida10 : 19 10 5 1 2 2 2 4 1 4 7 : 41100
aida11 : 21 10 2 1 2 3 3 3 2 3 7 : 39908
aida12 : 25 7 7 2 2 3 2 3 2 3 7 : 39756
aida13 : 13 7 3 4 2 5 2 3 2 3 7 : 39964
aida14 : 21 7 9 2 1 2 2 4 1 4 7 : 41116
aida15 : 23 6 2 3 0 4 3 3 2 3 7 : 39948
aida16 : 9 11 10 2 3 4 2 2 2 3 7 : 39452
Tom says Aida09 clock fail is ok as its status bit is "6".
The large white difference for FEE5 is known and has been determined to be ok, a post run investigation will be undertaken.
12:35 -
usbec ok.
Current ok
Stats good
Temps ok
Analysis ok R7_395. Dead time FEE1 still high 6.6%
13:33 -
usbec ok. Max implants ~ 1.8kHz
Current ok 006.850 uA 007.250 uA
Stats - Aida11 runing low < 5k was ~20k overnight
Temps ok
Analysis R7_415. Dead time FEE1 & FEE10 high 10%
14:00
usbec ok - ucesb1.png
Current ok - bias1.png
Stast - Aida11 low -
14:20 aida fee rebooted itself. A powercycle was performed. Upon reboot we are seeing extremely large amounts of noise in the FEEs. Looking at the waveforms we have very large 100kHz pick up in the FEEs. This has resulted in 50% deadtime in many FEEs including the p+n.
15:28 Because of extremely high rates across all FEEs have decided to do a powercycle. Before restarting the FEEs will give them a couple of minutes to cool.
18:00 Since the start of the shift we have been trying to recover the system froma large increase in noise following the crash at ~14:00.
During this time NH has entered the area and inspected the system and also grounded the AIDA snout. This provided us with some improvement on the noise.
The rates are still slightly above where we were before the crash but now appear stable. To counteract the dead time in the n+n strips we have raised the threshold to 0x64 for ASIC4 in all FEEs.
We are now running with around 10% deadtime on FEE4 and less elsewhere for n+n. For p+n most have zero dead time apart from FEE11 which is still noisy.
During the time we were trying to recover the system screenshots were taken of the waveforms. He it could be seen that the 100kHz noise was very apparent. Particularly in the n+n strips.
18:08 System wide checks all ok - bar some ADC but waveforms disabled
Statistics ok - 210516_1809_Stats
Temperatures ok - 210516_1809_Temp
Bias and leakage ok - 210516_1810_Bias
18:37 Performed an ASIC check and now the rates have dropped in all n+n strips. Currently very small amounts of dead time
18:40 Realised this was because it raised the threshold of all strips to 0x64 on the n+n side.
19:25 Removed S452 from 1e2.... drive. Before removing checked with Nic backed up to Lustre. Also verified four ourselves.
Now have around 4.2TB left which will provide around 80 hours of writing
20:16 We noticed iptraf was using around 30% CPU usage. We investigated whether it had any effect on the dead time but from what we have seen it has not.
20:57 System wide checks. Clock ok
Base Current Difference
aida05 fault 0x1552 : 0x1556 : 4
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Statistics ok - 210516_2056_Stats
Temp ok - 210516_2058_Temp
Bias and leakage current ok - 210516_2058_Bias
23:16 System wide checks:
Clock still ok
Base Current Difference
aida05 fault 0x1552 : 0x155a : 8
White Rabbit error counter test result: Passed 15, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida05 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 15, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Statistics - 210516_2315_Stats
Temperature - 210516_2316_Temp
Bias and leakage current ok - 210516_231 |
| Attachment 1: Screenshot_2021-05-16_Statistics_aidas-gsi(4).png
|
.png.png "Screenshot_2021-05-16_Statistics_aidas-gsi(4).png")
|
| Attachment 2: Screenshot_2021-05-16_Statistics_aidas-gsi(5).png
|
.png.png "Screenshot_2021-05-16_Statistics_aidas-gsi(5).png")
|
| Attachment 3: Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(4).png
|
.png.png "Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(4).png")
|
| Attachment 4: Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(5).png
|
.png.png "Screenshot_2021-05-16_Temperature_and_status_scan_aidas-gsi(5).png")
|
| Attachment 5: 1Statistics_aidas-gsi(6).png
|
.png.png "1Statistics_aidas-gsi(6).png")
|
| Attachment 6: 1Temperature_and_status_scan_aidas-gsi(6).png
|
.png.png "1Temperature_and_status_scan_aidas-gsi(6).png")
|
| Attachment 7: Bias1.png
|
|
| Attachment 8: 210516_0857_Temp.png
|
|
| Attachment 9: 210516_1554_Layout8.png
|
|
| Attachment 10: 210516_1555_Layout7.png
|
|
| Attachment 11: 210516_1600_Layout1.png
|
|
| Attachment 12: 210516_1601_Stats.png
|
|
| Attachment 13: 210516_1605_1606.png
|
|
| Attachment 14: 210516_1808_Stats.png
|
|
| Attachment 15: 210516_1809_Temp.png
|
|
| Attachment 16: 210516_1810_Bias.png
|
|
| Attachment 17: 210516_2056_Stats.png
|
|
| Attachment 18: 210516_2058_Bias.png
|
|
| Attachment 19: 210516_2313_Stats.png
|
|
| Attachment 20: 210516_2315_Temp.png
|
|
| Attachment 21: 210516_2316_Bias.png
|
|
|
235
|
Sat Apr 17 15:29:27 2021 |
MS | Saturday 17 April 16:00-20:00 2021 |
16:21
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bcf : 2
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Sat Apr 17 06:14:30 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
18:29
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida05 fault 0x1a52 : 0x1a53 : 1
aida06 fault 0x4f3e : 0x4f3f : 1
aida07 fault 0x3bcd : 0x3bd1 : 4
aida08 fault 0xc7c7 : 0xc7c8 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x0 : 0x1 : 1
aida12 fault 0x0 : 0x2 : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 21 3 4 4 1 3 1 3 3 3 6 : 36332
aida02 : 4 3 13 2 2 3 2 4 2 3 6 : 36152
aida03 : 12 7 5 2 2 3 2 4 2 3 6 : 36088
aida04 : 17 6 9 5 1 1 2 3 3 3 6 : 36452
aida05 : 26 8 0 3 1 1 2 3 3 3 6 : 36296
aida06 : 22 6 9 4 3 4 2 3 3 3 6 : 36952
aida07 : 17 7 3 1 2 3 3 2 3 3 6 : 36300
aida08 : 24 5 4 4 1 1 2 3 3 3 6 : 36360
aida09 : 15 11 3 2 1 3 1 3 3 3 6 : 36292
aida10 : 2 3 5 2 1 2 2 2 3 3 6 : 35824
aida11 : 2 2 0 0 3 2 2 4 2 3 6 : 35800
aida12 : 18 15 7 5 4 3 1 3 3 3 6 : 36688
|
| Attachment 1: 40.png
|
|
| Attachment 2: Screenshot_2021-04-17_Statistics_aidas-gsi(2).png
|
.png.png "Screenshot_2021-04-17_Statistics_aidas-gsi(2).png")
|
| Attachment 3: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png
|
.png.png "Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(2).png")
|
| Attachment 4: Screenshot_2021-04-17_Statistics_aidas-gsi(3).png
|
.png.png "Screenshot_2021-04-17_Statistics_aidas-gsi(3).png")
|
| Attachment 5: Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png
|
.png.png "Screenshot_2021-04-17_Temperature_and_status_scan_aidas-gsi(3).png")
|
| Attachment 6: 41.png
|
|
|
244
|
Mon Apr 19 06:46:15 2021 |
MS | Monday 19 April 07:47-12:00 2021 |
07:47
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c46 : 74
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0x35 : 43
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 18 10 7 1 3 2 4 2 3 6 : 36380
aida02 : 11 15 5 3 2 3 2 3 1 4 6 : 36692
aida03 : 15 11 13 3 4 3 3 3 2 3 6 : 36164
aida04 : 22 14 20 5 1 3 4 3 2 3 6 : 36456
aida05 : 7 7 5 2 1 2 3 4 2 3 6 : 36132
aida06 : 21 14 6 2 3 3 1 2 2 4 6 : 37028
aida07 : 11 8 7 4 2 3 2 2 4 3 6 : 37212
aida08 : 18 10 7 5 3 1 4 3 2 3 6 : 36072
aida09 : 24 23 18 3 2 3 2 4 2 3 6 : 36504
aida10 : 6 19 5 2 2 2 2 3 2 3 6 : 35520
aida11 : 22 15 7 2 1 3 2 3 2 3 6 : 35648
aida12 : 35 13 8 2 5 3 1 3 3 3 6 : 36724
At 8:30 DAQ and Temperature Scan stopped working properly.
11:00
It was noted that at the time that AIDA dropped out ~8:30 the beam also dropped out. It seems a bit of a coincidence
TD and MS powercycled the FEEs after the first crash and restarted MIDAS but not the merger. This recovered the FEEs but did not re-establish the links between the FEEs and the Merger.
Another powercycle was performed this time with a full reset of the merger and the links were restored. Upon restoring there was a large amount of noise in across all FEEs on average a 50% increase across all FEEs but in DSSD a factor of 4-6 increase was common.
In the waveforms large 100kHz transiets could be seen - attachment 4
A further power-cycle was performed by OH. The rates following this powercycle are better than just before the power cycle but have not recovered to pre-glitch levels.
The waveforms here are much improved though - attachment 5 and 6
11:45
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 12, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 1 2 2 1 2 4 2 4 2 3 7 : 40148
aida02 : 5 4 2 2 1 2 2 3 2 3 7 : 39380
aida03 : 3 2 1 1 1 3 1 4 2 3 7 : 39692
aida04 : 4 2 1 0 1 3 3 3 2 3 7 : 39664
aida05 : 2 7 2 0 2 2 3 4 2 3 7 : 40160
aida06 : 2 3 2 2 2 2 1 3 3 3 7 : 40192
aida07 : 23 7 2 0 0 3 2 4 2 3 7 : 39988
aida08 : 23 11 4 0 1 3 1 4 2 3 7 : 39860
aida09 : 1 3 1 1 2 4 1 3 1 4 7 : 40396
aida10 : 1 3 3 1 1 4 1 2 3 3 7 : 39852
aida11 : 3 0 2 3 1 2 3 2 2 3 7 : 39116
aida12 : 5 8 0 3 1 4 2 3 2 3 7 : 39668
|
| Attachment 1: Screenshot_2021-04-19_Statistics_aidas-gsi.png
|
|
| Attachment 2: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 3: 42.png
|
|
| Attachment 4: Screenshot_2021-04-19_Spectrum_Browser_aidas-gsi(1).png
|
.png.png "Screenshot_2021-04-19_Spectrum_Browser_aidas-gsi(1).png")
|
| Attachment 5: 210319_1054_Layout7.png
|
|
| Attachment 6: 210419_1054_Layout8.png
|
|
| Attachment 7: Screenshot_2021-04-19_Statistics_aidas-gsi(1).png
|
.png.png "Screenshot_2021-04-19_Statistics_aidas-gsi(1).png")
|
| Attachment 8: Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(1).png
|
.png.png "Screenshot_2021-04-19_Temperature_and_status_scan_aidas-gsi(1).png")
|
| Attachment 9: 43.png
|
|
|
256
|
Wed Apr 21 06:56:42 2021 |
MS | Wednesday 21 April 08:00-12:00 2021 |
8:00
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida05 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 0, Failed 12
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 12, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x15 : 21
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 19 6 7 2 2 4 3 2 2 3 7 : 39596
aida02 : 25 3 2 1 1 3 2 3 2 3 7 : 39548
aida03 : 28 5 3 1 2 2 2 3 2 3 7 : 39528
aida04 : 27 6 2 2 2 4 2 4 2 3 7 : 40316
aida05 : 9 9 4 3 1 3 2 3 3 3 7 : 40652
aida06 : 21 7 6 1 2 2 4 2 2 3 7 : 39564
aida07 : 20 7 5 0 1 2 2 3 2 3 7 : 39448
aida08 : 21 5 4 1 1 2 3 3 2 3 7 : 39708
aida09 : 1 3 1 3 0 2 1 4 2 3 7 : 39564
aida10 : 20 6 3 2 1 2 3 3 2 3 7 : 39728
aida11 : 18 4 2 1 2 2 2 2 2 3 7 : 38952
aida12 : 19 8 7 1 1 2 3 3 2 3 7 : 39772
From the merger statistics during the night there have been 1355757 TS errors in aida12. Corrigan said they have been quiet since around 3am though.
10:00
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida01 failed
FEE64 module aida02 failed
FEE64 module aida03 failed
FEE64 module aida04 failed
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida08 failed
FEE64 module aida09 failed
FEE64 module aida10 failed
FEE64 module aida11 failed
FEE64 module aida12 failed
Calibration test result: Passed 1, Failed 11
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 12, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x15 : 21
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 15 7 6 2 1 4 3 2 2 3 7 : 39508
aida02 : 1 2 1 0 1 2 2 4 2 3 7 : 39780
aida03 : 23 8 2 0 1 3 2 3 2 3 7 : 39548
aida04 : 23 9 1 2 3 3 2 4 2 3 7 : 40244
aida05 : 14 7 2 2 4 3 2 3 1 4 7 : 40784
aida06 : 18 5 5 2 2 2 3 3 2 3 7 : 39808
aida07 : 27 4 3 2 2 3 3 2 2 3 7 : 39420
aida08 : 23 6 3 2 2 2 3 3 2 3 7 : 39804
aida09 : 18 4 6 3 2 4 3 2 2 3 7 : 39592
aida10 : 6 9 5 3 2 3 2 3 2 3 7 : 39696
aida11 : 16 4 1 2 3 4 1 3 2 3 7 : 39536
aida12 : 18 5 7 1 0 3 3 3 2 3 7 : 39808
10:22
NH Noticed that the ASIC clocks had not been synchronised following the last AIDA restart. - attachment 7
As the beam is having issues they are not currently recording
I have now synchronised the ASIC clocks.
All ADC are now calibrated
11:06
Used the beam off time to change the drive writing to. Now writing to /media/ThirdDrive/TapeData/S460/R50
11:10
Beam is back they will start writing to file again
12:00
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit error counter test result: Passed 12, Failed 0
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x0 : 0x17 : 23
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 20 5 5 1 1 4 2 3 2 3 7 : 39720
aida02 : 16 8 4 1 0 2 1 4 2 3 7 : 39648
aida03 : 9 7 3 2 3 3 2 3 2 3 7 : 39692
aida04 : 8 7 5 2 3 5 3 2 2 3 7 : 39720
aida05 : 20 5 3 4 2 3 3 3 2 3 7 : 39976
aida06 : 25 8 0 3 1 2 3 3 2 3 7 : 39748
aida07 : 24 8 4 2 3 3 2 3 2 3 7 : 39776
aida08 : 22 9 0 2 2 2 3 3 2 3 7 : 39776
aida09 : 1 4 4 3 2 3 2 3 2 3 7 : 39620
aida10 : 13 4 4 3 2 3 1 3 2 3 7 : 39412
aida11 : 2 3 1 1 3 4 2 2 2 3 7 : 39184
aida12 : 16 10 7 2 1 3 1 3 2 3 7 : 39424
|
| Attachment 1: Screenshot_2021-04-21_Statistics_aidas-gsi.png
|
|
| Attachment 2: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 3: 46.png
|
|
| Attachment 4: Screenshot_2021-04-21_Statistics_aidas-gsi(1).png
|
.png.png "Screenshot_2021-04-21_Statistics_aidas-gsi(1).png")
|
| Attachment 5: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(1).png
|
.png.png "Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(1).png")
|
| Attachment 6: 47.png
|
|
| Attachment 7: Capture.PNG
|
|
| Attachment 8: Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(2).png
|
.png.png "Screenshot_2021-04-21_Temperature_and_status_scan_aidas-gsi(2).png")
|
| Attachment 9: Screenshot_2021-04-21_Statistics_aidas-gsi(2).png
|
.png.png "Screenshot_2021-04-21_Statistics_aidas-gsi(2).png")
|
| Attachment 10: 48.png
|
|
|
489
|
Wed Jun 22 07:12:46 2022 |
MS | Wednesday 22 June 8:00-16:00 |
Took over the night shift from Tom.
8:10 : attachments 1-3
10:00 : attachments 4-6
12:00 : attachments 7-9
The beam was off for around 20' starting at around 11:30-11:50 due to ion source issues.
12:19 OH Disabled all discriminators to reduce the data rate.
13:06 Analysis of R2_104 15.66% deadtime on AIDA04- attachment 10
14:00 : attachments 11-13
16:00 : attachments 14-16
|
| Attachment 1: Screenshot_from_2022-06-22_08-11-30.png
|
|
| Attachment 2: Screenshot_2022-06-22_at_08-10-58_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 3: Screenshot_2022-06-22_at_08-10-16_Statistics_aidas-gsi.png
|
|
| Attachment 4: Screenshot_2022-06-22_at_09-59-11_Statistics_aidas-gsi.png
|
|
| Attachment 5: Screenshot_2022-06-22_at_09-59-40_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 6: Screenshot_from_2022-06-22_10-00-13.png
|
|
| Attachment 7: Screenshot_2022-06-22_at_11-58-57_Statistics_aidas-gsi.png
|
|
| Attachment 8: Screenshot_2022-06-22_at_11-59-18_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 9: Screenshot_from_2022-06-22_11-59-55.png
|
|
| Attachment 10: R2_104_analysis.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 260553751 ( 867725.3 Hz)
Other data format: 1366249 ( 4550.0 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1464 ( 4.9 Hz)
RESUME: 1464 ( 4.9 Hz)
SYNC100: 33119 ( 110.3 Hz)
WR48-63: 33119 ( 110.3 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 1297083 ( 4319.7 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 518974 ( 1728.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 300.272 s
FEE elapsed dead time(s) elapsed idle time(s)
0 7.776 0.000
1 47.563 0.000
2 0.466 0.000
3 4.438 0.000
4 0.149 14.780
5 16.612 0.000
6 0.972 0.000
7 0.380 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 27837591 27876 0 0 187 187 3557 3557 0 20388 0 104437
1 57538232 16015 0 0 651 652 7356 7356 0 0 0 64114
2 23047755 331545 0 0 63 63 2949 2949 0 325521 0 64002
3 44324924 329582 0 0 146 145 5671 5671 0 317949 0 37242
4 11203550 2766 0 0 57 57 1326 1326 0 0 0 90620
5 28839527 7636 0 0 219 219 3599 3599 0 0 0 65648
6 37663072 317251 0 0 87 87 4772 4772 0 307533 0 63360
7 30099100 333578 0 0 54 54 3889 3889 0 325692 0 29551
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.320s ( 932.392 blocks/s, 58.275 Mb/s)
|
| Attachment 11: Screenshot_2022-06-22_at_14-01-58_Statistics_aidas-gsi.png
|
|
| Attachment 12: Screenshot_2022-06-22_at_14-02-23_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 13: Screenshot_from_2022-06-22_14-03-14.png
|
|
| Attachment 14: Screenshot_2022-06-22_at_15-57-21_Statistics_aidas-gsi.png
|
|
| Attachment 15: Screenshot_2022-06-22_at_15-57-42_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 16: Screenshot_from_2022-06-22_15-58-16.png
|
|
|
499
|
Sat Jun 25 06:55:07 2022 |
MS | Saturday 25 June 2022 8:00-16:00 |
Took over the night shift from Tom.
7:00 The beam is back.
8:00 : attachments 1-3
10:00 : attachments 4-6
12:00 : attachments 7-9
14:00 : attachments 10-12
16:00 : attachments 13-15
|
| Attachment 1: Screenshot_2022-06-25_at_08-01-16_Statistics_aidas-gsi.png
|
|
| Attachment 2: Screenshot_2022-06-25_at_08-01-37_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 3: Screenshot_from_2022-06-25_08-02-58.png
|
|
| Attachment 4: Screenshot_2022-06-25_at_10-00-41_Statistics_aidas-gsi.png
|
|
| Attachment 5: Screenshot_2022-06-25_at_10-01-03_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 6: Screenshot_from_2022-06-25_10-01-59.png
|
|
| Attachment 7: Screenshot_2022-06-25_at_12-00-54_Statistics_aidas-gsi.png
|
|
| Attachment 8: Screenshot_2022-06-25_at_12-01-17_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 9: Screenshot_from_2022-06-25_12-02-08.png
|
|
| Attachment 10: Screenshot_2022-06-25_at_14-07-36_Statistics_aidas-gsi.png
|
|
| Attachment 11: Screenshot_2022-06-25_at_14-07-59_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 12: Screenshot_from_2022-06-25_14-08-50.png
|
|
| Attachment 13: Screenshot_2022-06-25_at_15-56-41_Statistics_aidas-gsi.png
|
|
| Attachment 14: Screenshot_2022-06-25_at_15-56-57_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 15: Screenshot_from_2022-06-25_15-58-15.png
|
|
|
735
|
Thu Oct 30 10:06:21 2025 |
MP, NH | Setup and data taking with 207Bi |
- Initialised the Merger and Tape server
- loading of settings: EXPERIMENTS/AIDA/2025Oct29-16.11.06
- aida16 very noisy (attachments 3,4), slow comparator thresholds was risen to 500 keV, until reducing its deadtime to 0
- y FEEs disabled in the discriminator (example in attachment 5)
- Starting data taking with following details:
MIDAS data in: Bi207Centre
MBS data in /lustre/despec/aida_sourcetest_2025/raw started from lxg3138 on screen ('AIDAdaq') |
| Attachment 1: Screenshot_from_2025-10-30_11-07-55.png
|
|
| Attachment 2: Screenshot_from_2025-10-30_11-07-39.png
|
|
| Attachment 3: Screenshot_from_2025-10-30_11-07-29.png
|
|
| Attachment 4: Screenshot_from_2025-10-30_11-08-07.png
|
|
| Attachment 5: Screenshot_from_2025-10-30_11-10-06.png
|
|
|
736
|
Thu Oct 30 16:43:36 2025 |
MP, NH | Setup and data taking with 207Bi |
Data taking and DAQ stopped now (17.40)
| Quote: |
|
- Initialised the Merger and Tape server
- loading of settings: EXPERIMENTS/AIDA/2025Oct29-16.11.06
- aida16 very noisy (attachments 3,4), slow comparator thresholds was risen to 500 keV, until reducing its deadtime to 0
- y FEEs disabled in the discriminator (example in attachment 5)
- Starting data taking with following details:
MIDAS data in: Bi207Centre
MBS data in /lustre/despec/aida_sourcetest_2025/raw started from lxg3138 on screen ('AIDAdaq')
|
|
|
738
|
Tue Nov 4 16:34:47 2025 |
MP, NH | Bi207 source data, 700keV |
Some FEES were not acquiring data for some reason we could not understand. We lowered the thresholds to 200 keV, then rose them again to 700 keV. It seems they are now acquiring. We start data taking again.
| Quote: |
|
- Bias and basic checks: the pulser FWHM appears to be ~170, which is the same we have obtained at the end of last week.
- Initialised the Merger and Tape server
- loading of settings: EXPERIMENTS/AIDA/2025Nov03-16.21.44
- slow comparator thresholds set to 700 keV for all X FEES
- y FEEs disabled in the discriminator
- Starting data taking with following details:
MIDAS data in: Bi207Centre_700
MBS data in /lustre/despec/aida_sourcetest_2025/raw started from lxg3138 on screen ('AIDAdaq')
We take data overnight in this configuration and continue the grounding checks tomorrow
|
|
|
708
|
Tue Jul 22 17:00:26 2025 |
MP, CC | Status |
Test with shielded HV cables for both detectors.
The FWHM for ch1 is now 184 ch.
Yesterday cables for cards 5, 6, 10, 11, 12, 14, 16 were partially connected (only 1 out of 2 flat cables was in), now both are connected. |
| Attachment 1: Screenshot_from_2025-07-22_17-51-52.png
|
|
| Attachment 2: Screenshot_from_2025-07-22_17-59-20.png
|
|
|
707
|
Mon Jul 21 13:01:19 2025 |
MP | Status |
Status before grounding checks |
| Attachment 1: Screenshot_from_2025-07-21_14-00-23.png
|
|
| Attachment 2: Screenshot_from_2025-07-21_14-00-39.png
|
|
|
730
|
Wed Oct 29 08:12:35 2025 |
MP | Further setup for 207Bi test |
Inspection of side FEEs (2,6,4,8) and possible grounding improvements:
- all screws and nuts for copper connections were tightenedFee
- FEE 6 was isolated from the copper connection grounding FEE2 with insulating tape (attachment 1)
- A piece of tape was found below FEE6 (attachments 2,3) --> shall this be removed?
Bias and power on of FEES, preparation of two settings with lower thresholds:
- note: cannot calibrate ADC of aida15 even after trying multiple times (?)
- 500-keV thresholds settings saved as EXPERIMENTS/AIDA/2025Oct29-09.49.01
- 200-keV thresholds settings saved as EXPERIMENTS/AIDA/2025Oct29-09.55.31
|
| Attachment 1: IMG_5959.jpeg
|
|
| Attachment 2: IMG_5958.jpeg
|
|
| Attachment 3: IMG_5957.jpeg
|
|
| Attachment 4: Screenshot_from_2025-10-29_09-23-32.png
|
|
| Attachment 5: Screenshot_from_2025-10-29_09-32-52.png
|
|
|
737
|
Tue Nov 4 14:57:19 2025 |
MP | Bi207 source data, 700keV |
- Bias and basic checks: the pulser FWHM appears to be ~170, which is the same we have obtained at the end of last week.
- Initialised the Merger and Tape server
- loading of settings: EXPERIMENTS/AIDA/2025Nov03-16.21.44
- slow comparator thresholds set to 700 keV for all X FEES
- y FEEs disabled in the discriminator
- Starting data taking with following details:
MIDAS data in: Bi207Centre_700
MBS data in /lustre/despec/aida_sourcetest_2025/raw started from lxg3138 on screen ('AIDAdaq')
We take data overnight in this configuration and continue the grounding checks tomorrow |
| Attachment 1: Screenshot_from_2025-11-04_15-49-20.png
|
|
| Attachment 2: Screenshot_from_2025-11-04_15-33-11.png
|
|
| Attachment 3: Screenshot_from_2025-11-04_15-28-42.png
|
|
| Attachment 4: Screenshot_from_2025-11-04_15-28-35.png
|
|
|
739
|
Tue Nov 4 17:29:28 2025 |
MP | Bi207 source data, 700keV |
Anydesk has crashed. I am not able to set it up again. I switch off bias and fees
| Quote: |
|
Some FEES were not acquiring data for some reason we could not understand. We lowered the thresholds to 200 keV, then rose them again to 700 keV. It seems they are now acquiring. We start data taking again.
| Quote: |
|
- Bias and basic checks: the pulser FWHM appears to be ~170, which is the same we have obtained at the end of last week.
- Initialised the Merger and Tape server
- loading of settings: EXPERIMENTS/AIDA/2025Nov03-16.21.44
- slow comparator thresholds set to 700 keV for all X FEES
- y FEEs disabled in the discriminator
- Starting data taking with following details:
MIDAS data in: Bi207Centre_700
MBS data in /lustre/despec/aida_sourcetest_2025/raw started from lxg3138 on screen ('AIDAdaq')
We take data overnight in this configuration and continue the grounding checks tomorrow
|
|
|
|
742
|
Fri Nov 7 10:47:29 2025 |
MP | 207Bi source test - data taking |
DAQ stopped at ~17.30 due to anydesk failure
> Restart AIDA
>
> aida06 took three attempts to boot today :(
>
> All system checks OK, ASICs aligned
>
> Run DAQ without merger - looks okay, all FEEs working
>
> When connecting to merger aida15 drops connection, solution restart merger
>
> Aida13 not sending data - do an ASIC check/load
>
> Finally all DAQs send data, merger is happy with all 16 FEEs
>
> Open run R6 on disk |
|
240
|
Sun Apr 18 17:27:44 2021 |
ML-OH | system wide checks |
system wide checks at 6pm CET
##################################################
Difference noted from previous shift:
FPGS timestamp error 10 passed and 2 failed
##################################################
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC Calibration:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
White Rabbit checks:
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c11 : 21
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp errors checks:
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0xc : 2
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Memory information fro FEE64:
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 16 4 2 2 2 1 3 3 3 6 : 36332
aida02 : 19 8 9 3 2 2 2 4 2 3 6 : 36092
aida03 : 23 10 8 6 4 4 3 3 2 3 6 : 36332
aida04 : 27 4 8 0 1 3 4 3 2 3 6 : 36044
aida05 : 21 6 5 4 4 2 2 2 2 4 6 : 37204
aida06 : 23 11 5 0 3 4 1 3 3 3 6 : 36548
aida07 : 14 9 5 2 3 3 2 2 4 3 6 : 37200
aida08 : 26 8 7 2 2 1 4 3 2 3 6 : 35928
aida09 : 2 3 4 1 1 3 3 4 2 3 6 : 36160
aida10 : 16 9 2 5 1 1 3 4 2 3 6 : 36104
aida11 : 7 5 0 0 3 3 1 4 2 3 6 : 35716
aida12 : 9 7 3 2 4 4 1 3 3 3 6 : 36556
|
| Attachment 1: 18-04-2021_6pm_CAENHV.png
|
|
| Attachment 2: 18-04-2021_6pm_StatGoodEvent.png
|
|
| Attachment 3: 18-04-2021_6pm_ADCdataItem.png
|
|
| Attachment 4: 18-04-2021_6pm_WR28-47#4.png
|
|
| Attachment 5: 18-04-2021_6pm_WR48-63#5.png
|
|
| Attachment 6: 18-04-2021_6pm_PauseInfo.png
|
|
| Attachment 7: 18-04-2021_6pm_ResumeInfo.png
|
|
| Attachment 8: 18-04-2021_6pm_DiscInfo.png
|
|
| Attachment 9: 18-04-2021_6pm_Correlation#8.png
|
|
| Attachment 10: 18-04-2021_6pm_ucesb.png
|
|
| Attachment 11: 18-04-2021_645pm_tempScan.png
|
|
| Attachment 12: 18-04-2021_645pm_Spec1-8L.png
|
|
| Attachment 13: 18-04-2021_7pm_Spec1-8L_2.png
|
|
| Attachment 14: 18-04-2021_7pm_Spec1-8_layoutId4.png
|
|
| Attachment 15: 18-04-2021_7pm_Spec1-8H.png
|
|
| Attachment 16: 18-04-2021_645pm_SpecRate.png
|
|
| Attachment 17: 18-04-2021_730pm_AIDA_LC.png
|
|
|
241
|
Sun Apr 18 19:05:33 2021 |
ML-OH | All spectra reset |
All spectra reset at 8pm CET.
|
|
242
|
Sun Apr 18 21:43:01 2021 |
ML-OH | system wide checks |
New System wide check 10pm CET
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
ADC calibration check:
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida01 fault 0x7686 : 0x768a : 4
aida02 fault 0x941d : 0x9421 : 4
aida03 fault 0x7cd7 : 0x7cdb : 4
aida04 fault 0xb86d : 0xb871 : 4
aida05 fault 0x1a59 : 0x1a61 : 8
aida06 fault 0x4f45 : 0x4f4d : 8
aida07 fault 0x3bfc : 0x3c1d : 33
aida08 fault 0xc7ce : 0xc7d5 : 7
aida09 fault 0xb33b : 0xb33c : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp error checks:
Base Current Difference
aida09 fault 0x1 : 0x2 : 1
aida12 fault 0xa : 0xf : 5
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Memory from FEE64:
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 22 19 6 4 2 3 2 4 2 3 6 : 36304
aida02 : 23 9 3 3 1 3 1 4 2 3 6 : 35828
aida03 : 15 5 6 3 2 3 3 3 2 3 6 : 35876
aida04 : 20 8 18 1 1 3 4 3 2 3 6 : 36240
aida05 : 21 7 1 2 1 2 4 3 2 3 6 : 35868
aida06 : 18 8 4 1 3 4 1 3 3 3 6 : 36520
aida07 : 4 9 3 4 2 3 2 2 3 3 6 : 36104
aida08 : 15 5 2 4 2 1 3 4 2 3 6 : 36100
aida09 : 2 23 8 1 2 3 2 4 2 3 6 : 36192
aida10 : 10 7 8 5 2 1 3 3 2 3 6 : 35712
aida11 : 1 9 5 1 2 3 1 4 2 3 6 : 35772
aida12 : 14 8 6 3 3 4 1 3 3 3 6 : 36600 |
| Attachment 1: 18-04-2021_10pm_HV.png
|
|
| Attachment 2: 18-04-2021_10pm_StatGoodEvt.png
|
|
| Attachment 3: 18-04-2021_10pm_PAUSE.png
|
|
| Attachment 4: 18-04-2021_10pm_RESUME.png
|
|
| Attachment 5: 18-04-2021_10pm_ADCdataItem.png
|
|
| Attachment 6: 18-04-2021_10pm_DISC.png
|
|
| Attachment 7: 18-04-2021_10pm_correlations.png
|
|
| Attachment 8: 18-04-2021_10pm_SpecRate.png
|
|
| Attachment 9: 18-04-2021_10pm_Spec1-8H.png
|
|
| Attachment 10: 18-04-2021_10pm_Spec1-8HlayoutID6.png
|
|
| Attachment 11: 18-04-2021_10pm_Spec1-8L.png
|
|
| Attachment 12: 18-04-2021_10pm_Spec1-8L_layoutid4.png
|
|
| Attachment 13: 18-04-2021_10pm_Spec1-8W.png
|
|
| Attachment 14: 18-04-2021_10pm_SpecLayoutID8.png
|
|
| Attachment 15: 18-04-2021_10pm_WR28-47.png
|
|
| Attachment 16: 18-04-2021_10pm_WR48-63.png
|
|
| Attachment 17: 18-04-2021_10pm_temp.png
|
|
| Attachment 18: 18-04-2021_10pm_ucesb_monitor.png
|
|
| Attachment 19: 18-04-2021_10pm-AIDA_Alerting_-_Grafana.png
|
|
| Attachment 20: 18-04-2021_10pm_NewMerger.png
|
|
|
228
|
Fri Apr 16 05:13:09 2021 |
ML | system wide checks |
system wide checks:
Done around 4am.
#################################
To be noted:
-1- White rabbit and FPGA checks failed for 1 FEE and passed for 11
-2- No waveform for AIDA09 in 1.8.L spectrum
-3- AIDA 7 is red in the NewMerger control window (?)
#################################
Details:
Clock status test result: Passed 12, Failed 0
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
Calibration test result: Passed 12, Failed 0
White Rabbit checks:
Base Current Difference
aida07 fault 0xfa8c : 0xfb37 : 171
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA timestamp check:
Base Current Difference
aida07 fault 0x0 : 0xe : 14
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Memory check:
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 45 9 7 3 4 3 2 3 3 3 6 : 36940
aida02 : 23 9 6 7 4 3 2 3 3 3 6 : 36964
aida03 : 40 8 5 4 5 3 2 3 3 3 6 : 36976
aida04 : 37 13 7 3 4 3 4 2 3 3 6 : 36940
aida05 : 35 11 6 3 4 4 2 3 2 4 6 : 38052
aida06 : 42 8 2 3 4 4 2 3 3 3 6 : 36968
aida07 : 42 10 5 5 5 3 2 3 3 3 6 : 37032
aida08 : 40 8 1 6 3 4 2 3 3 3 6 : 36976
aida09 : 18 18 14 8 5 5 5 4 3 2 6 : 36728
aida10 : 1 8 9 6 3 3 1 4 2 3 6 : 36052
aida11 : 24 11 14 6 2 2 4 2 3 3 6 : 36824
aida12 : 17 13 8 5 5 4 2 3 2 3 6 : 36108 |
| Attachment 1: 16-04-2021_3h45am_HV.png
|
|
| Attachment 2: 48_temperature.png
|
|
| Attachment 3: 16-04-2021_3h51_StatGoodEvt.png
|
|
| Attachment 4: 16-04-2021_3h53_StatADCdataItem.png
|
|
| Attachment 5: 16-04-2021_3h58_StatWR#4.png
|
|
| Attachment 6: 16-04-2021_4h04_StatWR#5.png
|
|
| Attachment 7: 16-04-2021_3h59_StatPause#2.png
|
|
| Attachment 8: 16-04-2021_4h01_StatResume#3.png
|
|
| Attachment 9: 16-04-2021_4h02_StatCorrelation#8.png
|
|
| Attachment 10: 16-04-2021_4h06_SpecRate.png
|
|
| Attachment 11: 16-04-2021_4h08_SpecRate2.png
|
|
| Attachment 12: 16-04-2021_4h17_Spec1.8L.png
|
|
| Attachment 13: 16-04-2021_4h29_Spec1.8L_id4.png
|
|
| Attachment 14: 16-04-2021_4h20_Spec1.8W_id7.png
|
|
| Attachment 15: 25_Spec1.8W_id8.png
|
|
| Attachment 16: 16-04-2021_4h48_NewMergerControl.png
|
|