| ID |
Date |
Author |
Subject |
|
626
|
Thu May 2 23:08:43 2024 |
TD | Friday 3 May |
00.05 FEE64 temperatures OK
ADC data item stats OK
09.31 FEE64 temperatures OK
ADC data item stats OK
13.15 FEE64 temperatures OK
ADC data item stats OK
9/16 <20k max aida16 164k |
|
625
|
Thu May 2 00:16:22 2024 |
TD, NH | Thursday 2 May |
01.15 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 no data
6/16 < 20k
10.25 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 restarted
7/16 < 20k, max 220k aida16
all histograms zero'd
13.20 per p+n FEE64 1.8.L spectra - attachment 1
aida09 pulser peak width 59 ch FWHM
FEE64 temperatures OK
ADC data item stats
aida04 no data
5/16 < 20k, max 332k aida08
14.37
Moved Time Machine delayed scaler from aida04 to aida07
aida04 no data
Reset merger etc
aida05 no data
Reset merger etc
All FEEs with data again
S4 Ambient Environment: 26.6 C / 30.4% RH / Td=7.8 C
17.26 FEE64 temperatures OK
ADC data item stats OK
19.28 FEE64 temperatures OK
ADC data item stats
aida02 no data - DAQ reset
|
| Attachment 1: Screenshot_from_2024-05-02_13-21-07.png
|
|
| Attachment 2: Screenshot_from_2024-05-02_13-22-16.png
|
|
| Attachment 3: Screenshot_from_2024-05-02_13-22-43.png
|
|
|
624
|
Wed May 1 08:59:15 2024 |
TD | Wednesday 1 May |
04.55 FEE64 temperatures OK
ADC data item stats OK *except* aida04 no data
TapeServer no storage mode
09.55 FEE64 temperatures OK
ADC data item stats OK *except* aida02 & aida04 no data
DSSSD bias and leakage current - Grafana - attachment 1
19.28 FEE64 temperatures OK
ADC data item stats
aida02 & aida04 no data
noted increase in rates for aid08 and aida16
5/16 < 20k |
| Attachment 1: Capture.PNG
|
|
|
623
|
Tue Apr 30 09:06:45 2024 |
TD, NH | Tuesday 30 April |
10.00 DSSSD bias & leakasge current - Grafana - attachments 1-2
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 4
9/16 FEE64s < 20k
per FEE64 Rate spectra - attachment 5
Merger etc - attachments 6-7
aida02, aida04 inactive
11.20 DAQ restarted
aida02 and aida04 now producing data
11.30 analysis data file R22_313
all deadtimes <1%
no timewarps
HEC data item rate 560Hz (all from one channel of aida08 asic #3)
aida03 & aida04 MBS rates c. 24kHz
14.32 FEE64 temperatures OK
ADC data item stats OK
all FEE64s producing data
14.43 all histograms & stats zero'd
[NH checks here]
21.26 aida04 stopped forwarding to merger
the "quick fix" (stop merger, rerun) worked
MBS is having huge issues on DESPEC side, AIDA not included
Decide to stop taking data to disk for now (no storage mode on)
Keeping running (incl. to AIDA FDR MBS) just to get an idea of stability
System Wide Checks
clocks PASSED
ADC IGNORED
WR:
Base Current Difference
aida02 fault 0xe4d8 : 0x65c0 : -32536
aida03 fault 0x2371 : 0x2380 : 15
aida04 fault 0xa210 : 0xaf58 : 3400
aida06 fault 0x3624 : 0x3626 : 2
White Rabbit error counter test result: Passed 12, Failed 4
FPGA:
Base Current Difference
aida02 fault 0xaa3b : 0xc455 : 6682
aida04 fault 0x5a4 : 0x6ba : 278
aida06 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 13, Failed 3
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Module aida04 LMK PLL channel 1 lock transition counter has incremented
Module aida04 LMK PLL channel 2 lock transition counter has incremented
PLL lock counter test result: Passed 15, Failed 1
Baselines reset
22.50 FEE64 temps ok
merger ok
all links active
ADC data item stats ok |
| Attachment 1: Screenshot_from_2024-04-30_10-05-08.png
|
|
| Attachment 2: Screenshot_from_2024-04-30_10-05-37.png
|
|
| Attachment 3: Screenshot_from_2024-04-30_10-01-32.png
|
|
| Attachment 4: Screenshot_from_2024-04-30_10-02-03.png
|
|
| Attachment 5: Screenshot_from_2024-04-30_10-03-26.png
|
|
| Attachment 6: Screenshot_from_2024-04-30_10-03-58.png
|
|
| Attachment 7: R22_313
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 246802750 ( 787077.4 Hz)
Other data format: 15117251 ( 48210.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 802 ( 2.6 Hz)
RESUME: 802 ( 2.6 Hz)
SYNC100: 33168 ( 105.8 Hz)
WR48-63: 33168 ( 105.8 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 15049311 ( 47993.7 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 182597 ( 582.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 313.569 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 0.000
1 0.000 0.000
2 0.000 0.000
3 0.410 0.000
4 0.000 0.000
5 0.208 0.000
6 0.000 0.000
7 1.577 0.000
8 0.000 0.000
9 0.000 0.000
10 1.890 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 1.784 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 4403022 10517 0 0 0 0 556 556 0 9405 0 0
1 4797622 11663 0 0 0 0 607 607 0 10449 0 0
2 9960742 7551016 0 0 0 0 2246 2246 0 7546524 0 0
3 26989916 7491649 0 0 64 64 4294 4294 0 7482933 0 0
4 2737535 758 0 0 0 0 379 379 0 0 0 0
5 21805196 5662 0 0 70 70 2761 2761 0 0 0 0
6 6978804 1886 0 0 0 0 943 943 0 0 0 0
7 53329550 13820 0 0 126 126 6784 6784 0 0 0 182597
8 479454 122 0 0 0 0 61 61 0 0 0 0
9 5592721 1538 0 0 0 0 769 769 0 0 0 0
10 39929583 11138 0 0 441 441 5128 5128 0 0 0 0
11 2351324 636 0 0 0 0 318 318 0 0 0 0
12 5309169 1226 0 0 0 0 613 613 0 0 0 0
13 3913898 1006 0 0 0 0 503 503 0 0 0 0
14 2364855 578 0 0 0 0 289 289 0 0 0 0
15 55859359 14036 0 0 101 101 6917 6917 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 32.668s ( 979.553 blocks/s, 61.222 Mb/s)
|
|
622
|
Mon Apr 29 08:11:16 2024 |
TD | Monday 29 April |
08.52 Beam off
09.11 New data file R22
Monitor accumulated implanted activity in bPlas and DSSSDs
ASIC settings 2024Apr26-01.29.53
DSSSD bias & leakage current OK - attachments 1-3
FEE64 temperatures OK - attachment 4
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 6
Merger etc - attachment 7
analysis data file R22_19 - attachment 8
all deadtimes 0.3%, or less
no timewarps
HEC data item rate 390Hz
10.00 all histograms & stats zero'd
13.24 DSSSD bias & leakage current OK - attachments 9-10
FEE64 temperatures OK - attachment 11
ADC data item stats - attachment 12
per FEE64 Rate spectra - attachment 13
per p+n FEE64 1.8.L spectra - attachments 14-16
common x/y axes - peak height proxy for width
aida09 pulser peak width 58 ch FWHM (~41keV FWHM) cf. https://elog.ph.ed.ac.uk/DESPEC/585 attachment 38
electronic noise today very similar to electronic noise Monday 22 April
per FEE64 1.8.W spectra - 20us FSR - attachments 17-19
Merger etc - attachment 20
analysis data file R22_60 - attachment 21
max deadtime 1.7% (aida11), all others <1%
no timewarps
HEC data item rate 710Hz
14.13 ucesb - attachment 22
14.15 aida04 not producing data
14.33 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
15.27 *all* FEE64s stop producing data - reason unknown
16.15 DAQ reset per Elog 618 https://elog.ph.ed.ac.uk/DESPEC/618
16.29 DSSSD bias & leakage current OK - attachments 23-24
FEE64 temperatures OK - attachment 25
ADC data item stats - attachment 26
per FEE64 Rate spectra - attachment 27
Merger etc - attachment 28
20.45 DSSSD bias & leakage current OK - attachments 29-30
FEE64 temperatures OK - attachment 32
ADC data item stats - attachment 31
per FEE64 Rate spectra - attachment 33
Merger etc - attachment 34
20.54 analysis data file R22_145 - attachment 35
max deadtime 1.5% (aida11), all others <1%
no timewarps
HEC data item rate 775Hz
03.45 DSSSD bias & leakage current OK - attachments 36-37
FEE64 temperatures OK - attachment 38
ADC data item stats - attachment 39
per FEE64 Rate spectra - attachment 40
Merger etc - attachment 41 |
| Attachment 1: Screenshot_from_2024-04-29_09-13-12.png
|
|
| Attachment 2: Screenshot_from_2024-04-29_09-13-37.png
|
|
| Attachment 3: Screenshot_from_2024-04-29_09-13-56.png
|
|
| Attachment 4: Screenshot_from_2024-04-29_09-14-31.png
|
|
| Attachment 5: Screenshot_from_2024-04-29_09-15-05.png
|
|
| Attachment 6: Screenshot_from_2024-04-29_09-15-48.png
|
|
| Attachment 7: Screenshot_from_2024-04-29_09-17-25.png
|
|
| Attachment 8: R22_19
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 245162657 ( 661369.3 Hz)
Other data format: 16757344 ( 45205.9 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 432 ( 1.2 Hz)
RESUME: 432 ( 1.2 Hz)
SYNC100: 33381 ( 90.1 Hz)
WR48-63: 33381 ( 90.1 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 16689718 ( 45023.4 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 145285 ( 391.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 370.689 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 0.000
1 0.000 0.000
2 0.000 0.000
3 0.421 0.000
4 0.000 0.000
5 0.062 0.000
6 0.000 0.000
7 1.097 0.000
8 0.000 0.000
9 0.000 0.000
10 1.032 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 1.195 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 4689453 12333 0 0 0 0 606 606 0 11121 0 151
1 6714412 1102163 0 0 0 0 995 995 0 1100173 0 247
2 9093299 6196435 0 0 0 0 2024 2024 0 6192387 0 151
3 31978220 6022790 0 0 67 67 4731 4731 0 6013194 0 212
4 3345145 1695016 0 0 0 0 641 641 0 1693734 0 75
5 18560198 1684371 0 0 12 12 2619 2619 0 1679109 0 279
6 7471590 2034 0 0 0 0 1017 1017 0 0 0 150
7 51047482 13240 0 0 105 105 6515 6515 0 0 0 143158
8 1099097 280 0 0 0 0 140 140 0 0 0 152
9 7056038 1776 0 0 0 0 888 888 0 0 0 136
10 35026945 9268 0 0 144 144 4490 4490 0 0 0 97
11 2488733 676 0 0 0 0 338 338 0 0 0 118
12 6088987 1632 0 0 0 0 816 816 0 0 0 53
13 5057582 1278 0 0 0 0 639 639 0 0 0 108
14 2541330 642 0 0 0 0 321 321 0 0 0 127
15 52904146 13410 0 0 104 104 6601 6601 0 0 0 71
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 33.137s ( 965.696 blocks/s, 60.356 Mb/s)
|
| Attachment 9: Screenshot_from_2024-04-29_13-24-45.png
|
|
| Attachment 10: Screenshot_from_2024-04-29_13-25-48.png
|
|
| Attachment 11: Screenshot_from_2024-04-29_13-26-19.png
|
|
| Attachment 12: Screenshot_from_2024-04-29_13-26-51.png
|
|
| Attachment 13: Screenshot_from_2024-04-29_13-27-33.png
|
|
| Attachment 14: Screenshot_from_2024-04-29_13-28-45.png
|
|
| Attachment 15: Screenshot_from_2024-04-29_13-29-15.png
|
|
| Attachment 16: Screenshot_from_2024-04-29_13-30-17.png
|
|
| Attachment 17: Screenshot_from_2024-04-29_13-32-51.png
|
|
| Attachment 18: Screenshot_from_2024-04-29_13-33-20.png
|
|
| Attachment 19: Screenshot_from_2024-04-29_13-34-07.png
|
|
| Attachment 20: Screenshot_from_2024-04-29_13-34-32.png
|
|
| Attachment 21: R22_60
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 258414198 ( 779475.1 Hz)
Other data format: 3505804 ( 10574.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 880 ( 2.7 Hz)
RESUME: 880 ( 2.7 Hz)
SYNC100: 33235 ( 100.2 Hz)
WR48-63: 33235 ( 100.2 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 3437574 ( 10369.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 235960 ( 711.7 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 331.523 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.095 0.000
1 0.000 0.000
2 0.000 0.000
3 0.000 0.000
4 0.000 0.000
5 0.217 0.000
6 0.140 0.000
7 2.432 0.000
8 0.000 0.000
9 0.011 0.000
10 5.658 0.000
11 0.000 0.000
12 0.051 0.000
13 0.000 0.000
14 0.000 0.000
15 2.652 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 4927110 11209 0 0 4 4 628 628 0 9945 0 39
1 5556527 109296 0 0 0 0 741 741 0 107814 0 59
2 10335604 3081300 0 0 0 0 1668 1668 0 3077964 0 29
3 0 0 0 0 0 0 0 0 0 0 0 0
4 3176904 121984 0 0 0 0 428 428 0 121128 0 19
5 28235398 128309 0 0 73 73 3720 3720 0 120723 0 52
6 7534053 1958 0 0 2 2 977 977 0 0 0 22
7 65092755 17088 0 0 173 173 8371 8371 0 0 0 235567
8 631770 138 0 0 0 0 69 69 0 0 0 28
9 6806363 1686 0 0 1 1 842 842 0 0 0 28
10 44391302 12266 0 0 505 505 5628 5628 0 0 0 16
11 2632024 670 0 0 0 0 335 335 0 0 0 21
12 5950079 1370 0 0 1 1 684 684 0 0 0 14
13 4496081 1136 0 0 0 0 568 568 0 0 0 29
14 2644608 652 0 0 0 0 326 326 0 0 0 25
15 66003620 16742 0 0 121 121 8250 8250 0 0 0 12
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 37.793s ( 846.718 blocks/s, 52.920 Mb/s)
|
| Attachment 22: Capture.PNG
|
|
| Attachment 23: Screenshot_from_2024-04-29_16-29-34.png
|
|
| Attachment 24: Screenshot_from_2024-04-29_16-29-54.png
|
|
| Attachment 25: Screenshot_from_2024-04-29_16-30-39.png
|
|
| Attachment 26: Screenshot_from_2024-04-29_16-31-01.png
|
|
| Attachment 27: Screenshot_from_2024-04-29_16-31-51.png
|
|
| Attachment 28: Screenshot_from_2024-04-29_16-32-12.png
|
|
| Attachment 29: Screenshot_from_2024-04-29_20-45-26.png
|
|
| Attachment 30: Screenshot_from_2024-04-29_20-46-25.png
|
|
| Attachment 31: Screenshot_from_2024-04-29_20-46-52.png
|
|
| Attachment 32: Screenshot_from_2024-04-29_20-47-20.png
|
|
| Attachment 33: Screenshot_from_2024-04-29_20-48-08.png
|
|
| Attachment 34: Screenshot_from_2024-04-29_20-48-56.png
|
|
| Attachment 35: R22_145
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 247656598 ( 992529.4 Hz)
Other data format: 14263402 ( 57163.2 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 881 ( 3.5 Hz)
RESUME: 881 ( 3.5 Hz)
SYNC100: 32930 ( 132.0 Hz)
WR48-63: 32930 ( 132.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 14195780 ( 56892.2 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 193328 ( 774.8 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 249.521 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.003 0.000
1 0.000 0.000
2 0.000 0.000
3 0.318 0.000
4 0.000 0.000
5 1.537 0.000
6 0.000 0.000
7 1.906 0.000
8 0.000 0.000
9 0.000 0.000
10 3.841 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 1.886 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 6683029 9310 0 0 5 5 906 906 0 7488 0 74
1 5367157 1313019 0 0 0 0 827 827 0 1311365 0 63
2 11584446 4358913 0 0 0 0 1980 1980 0 4354953 0 37
3 22834625 4167004 0 0 37 37 3370 3370 0 4160190 0 57
4 3277605 2130735 0 0 0 0 657 657 0 2129421 0 19
5 24294581 2239601 0 0 194 194 3425 3425 0 2232363 0 49
6 6724993 1760 0 0 0 0 880 880 0 0 0 28
7 50507044 12910 0 0 126 126 6329 6329 0 0 0 192806
8 656116 188 0 0 0 0 94 94 0 0 0 10
9 5715462 1410 0 0 0 0 705 705 0 0 0 12
10 39787841 10690 0 0 431 431 4914 4914 0 0 0 4
11 3128950 784 0 0 0 0 392 392 0 0 0 67
12 4910404 1244 0 0 0 0 622 622 0 0 0 13
13 3786534 940 0 0 0 0 470 470 0 0 0 59
14 2967123 782 0 0 0 0 391 391 0 0 0 2
15 55430688 14112 0 0 88 88 6968 6968 0 0 0 28
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 33.727s ( 948.807 blocks/s, 59.300 Mb/s)
|
| Attachment 36: Screenshot_from_2024-04-30_03-45-13.png
|
|
| Attachment 37: Screenshot_from_2024-04-30_03-46-07.png
|
|
| Attachment 38: Screenshot_from_2024-04-30_03-46-43.png
|
|
| Attachment 39: Screenshot_from_2024-04-30_03-47-10.png
|
|
| Attachment 40: Screenshot_from_2024-04-30_03-47-40.png
|
|
| Attachment 41: Screenshot_from_2024-04-30_03-48-26.png
|
|
|
621
|
Sun Apr 28 22:52:06 2024 |
Marc | 0:00-08:00 Monday 29 April |
Just another manic magic Monday, ....
Taking over Betool's AIDA shift. All is good.
This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.
01:15: AIDA04 is down again. DAQ shift crew informed.
01:27: AIDA04 is back and shift crew has been informed.
01:30 Checks:
Bias and leakage current - Attachment 1-2
Temperatures - Attachment 3
ADC Data item - Attachment 4
Rate Spectra - Attachment 5
UCESB - Attachment 6
Timestamp checks - Attachement 7
06:00 It has run very smoothly.
|
| Attachment 1: Screenshot_from_2024-04-29_01-33-27.png
|
|
| Attachment 2: Screenshot_from_2024-04-29_01-34-37.png
|
|
| Attachment 3: Screenshot_from_2024-04-29_01-36-12.png
|
|
| Attachment 4: Screenshot_from_2024-04-29_01-36-41.png
|
|
| Attachment 5: Screenshot_from_2024-04-29_01-37-56.png
|
|
| Attachment 6: Screenshot_from_2024-04-29_01-32-39.png
|
|
| Attachment 7: Screenshot_from_2024-04-29_01-42-58.png
|
|
|
620
|
Sun Apr 28 20:35:57 2024 |
Betool Alayed | 16:0-00:00 28 April 2024 |
21:30 screenshots
and
23:24 screenshots |
| Attachment 1: Screenshot_from_2024-04-28_21-30-19.png
|
|
| Attachment 2: Screenshot_from_2024-04-28_21-31-11.png
|
|
| Attachment 3: Screenshot_from_2024-04-28_21-32-07.png
|
|
| Attachment 4: Screenshot_from_2024-04-28_21-32-51.png
|
|
| Attachment 5: Screenshot_from_2024-04-28_21-35-10.png
|
|
| Attachment 6: Screenshot_from_2024-04-28_23-35-25.png
|
|
| Attachment 7: Screenshot_from_2024-04-28_23-34-31.png
|
|
| Attachment 8: Screenshot_from_2024-04-28_23-33-42.png
|
|
| Attachment 9: Screenshot_from_2024-04-28_23-32-28.png
|
|
| Attachment 10: Screenshot_from_2024-04-28_23-31-51.png
|
|
|
619
|
Sun Apr 28 16:26:47 2024 |
Betool Alayed | 16:00-00:00 Sunday 28 April |
19:24 pm screenshots
and
17:18 pm screenshots |
| Attachment 1: Screenshot_from_2024-04-28_17-18-38.png
|
|
| Attachment 2: Screenshot_from_2024-04-28_17-13-13.png
|
|
| Attachment 3: Screenshot_from_2024-04-28_17-11-48.png
|
|
| Attachment 4: Screenshot_from_2024-04-28_17-10-32.png
|
|
| Attachment 5: Screenshot_from_2024-04-28_17-04-10.png
|
|
| Attachment 6: Screenshot_from_2024-04-28_19-18-42.png
|
|
| Attachment 7: Screenshot_from_2024-04-28_19-20-10.png
|
|
| Attachment 8: Screenshot_from_2024-04-28_19-22-19.png
|
|
| Attachment 9: Screenshot_from_2024-04-28_19-23-19.png
|
|
| Attachment 10: Screenshot_from_2024-04-28_19-24-40.png
|
|
|
618
|
Sun Apr 28 16:07:59 2024 |
TD | [How To] Recover DAQ when one FEE64 stops sending data - short(er) version |
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ
1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'
2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
terminal window showing the startup will popup
3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']
4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
all data links will be inactive (red)
5) Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']
6) Start DAQ
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
readout times will be displayed for each FEE64
check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support
8) Stop DAQ
Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ has *actually* stopped - if not repeat 'STOP
9) Enable data transfer to Merger and TapeServer and start DAQ
Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
10) Check data is being transferred to Merger and TapeServer
Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)
Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
you should see 'Blocks written' and 'kb written' counters increase
11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]
12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.
If not, please request expert assistance. |
|
617
|
Sun Apr 28 15:49:58 2024 |
TD | DAQ restart short(er) cut |
16.45 aida04 stopped producing data
DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
aida04 did not stop, all other FEE64s stopped
restart Merger (workspace 3, display 2, select 'NewMerger' icon)
SETUP Merger (workspace 3, display 1, browser 'NewMerger' tab)
GO Merger (workspace 3, display 1, browser 'NewMerger' tab)
aida04 now stops
disable *all* #1 data transfer
DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
check WR timestamps OK (workspace 2, display 1, GSI WR Timestamp Control)
DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
enable *all* #1 data transfer
DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
No reboot or power cycle required. Just clear the Merger data links, restart DAQ.
|
|
616
|
Sun Apr 28 15:09:34 2024 |
Betool Alayed | 16:0-00:00 28 April 2024 |
16:00 aida04 not producing data
power cycle all FEE64s to recover DAQ
16.20 analysis data file R21_668
max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%
no timewarps
HEC data item rate 1.9kHz
|
| Attachment 1: R21_668.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 256513573 ( 1160512.7 Hz)
Other data format: 5406427 ( 24459.6 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1079 ( 4.9 Hz)
RESUME: 1078 ( 4.9 Hz)
SYNC100: 32824 ( 148.5 Hz)
WR48-63: 32824 ( 148.5 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 5338622 ( 24152.9 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 440937 ( 1994.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 221.035 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.085 0.000
1 19.736 0.000
2 0.053 0.000
3 36.607 0.000
4 0.854 0.000
5 4.747 0.000
6 0.000 0.000
7 1.845 0.000
8 0.000 0.000
9 0.000 0.000
10 0.661 0.000
11 0.020 0.000
12 0.004 0.000
13 0.000 0.000
14 0.003 0.000
15 1.247 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 8138410 8624 0 0 12 12 985 985 0 6630 0 38979
1 27142829 245404 0 0 242 242 3461 3461 0 237998 0 64179
2 11096772 2350080 0 0 8 8 1768 1768 0 2346528 0 26862
3 37888832 1835283 0 0 304 303 4762 4762 0 1825152 0 18340
4 9329455 413600 0 0 28 28 1212 1212 0 411120 0 29227
5 23846951 517930 0 0 167 167 3201 3201 0 511194 0 24385
6 5635241 1420 0 0 0 0 710 710 0 0 0 8053
7 40074524 10318 0 0 93 93 5066 5066 0 0 0 120554
8 3088839 754 0 0 0 0 377 377 0 0 0 14672
9 5102920 1254 0 0 0 0 627 627 0 0 0 6080
10 25963054 6756 0 0 140 140 3238 3238 0 0 0 2640
11 5980678 1458 0 0 5 5 724 724 0 0 0 52714
12 5769255 1548 0 0 2 2 772 772 0 0 0 8398
13 3930590 926 0 0 0 0 463 463 0 0 0 9544
14 5399163 1328 0 0 1 1 663 663 0 0 0 5692
15 38126060 9744 0 0 77 77 4795 4795 0 0 0 10618
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.004s ( 941.068 blocks/s, 58.817 Mb/s)
|
|
615
|
Sun Apr 28 06:57:59 2024 |
Magda Satrazani | 08:00-16:00 Sunday 28 April |
Took over the shift from Marc.
10.51 Added screenshots to illustrate erroneous data from Grafana (AIDA) - attachments 18-19
Compare to the CAEN HV control terminal (attachment 15) which shows the expected values. Have email'd Nic & Helena. TD
10.55 Noticed that the CAEN HV control terminal is showing the corerct data but on the wrong line of the display (attachment 15). Enter commands q (quit) and d (display) to restart display - the correct data now appears on the correct line (attachment 20).
Grafana (AIDA) is now reporting data correctly too (attachment 21).
Summary - CAEN HV terminal display issue ( c. 10.05 - 10.55 )
Note the correct data/values continued to be shown. Leakage current shown correctly by both Grafana and CAEN HV control throughout.
Conclude this was a display issue - not a change in the operating parameters of the CAEN N1419ET.. Data should be OK.
14.45 analysis data file R21_500 - attachment 36
max deadtime 22% (aida11), 16% (aida04), 7% (aida02), 6% (aida08), 6% (aida06) , all others < 1%
no timewarps
HEC data item rate 2.7kHz
|
| Attachment 1: Screenshot_2024-04-28_at_08-00-09_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 2: Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
|
|
| Attachment 3: leakage_current.png
|
|
| Attachment 4: Screenshot_2024-04-28_at_08-05-38_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 5: Screenshot_2024-04-28_at_08-08-14_ucesb.png
|
|
| Attachment 6: Screenshot_2024-04-28_at_08-09-21_AIDA_-_Grafana.png
|
|
| Attachment 7: Screenshot_2024-04-28_at_08-11-20_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 8: Screenshot_2024-04-28_at_10-01-48_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 9: Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
|
|
| Attachment 10: Screenshot_2024-04-28_at_10-02-44_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 11: Screenshot_2024-04-28_at_10-03-40_AIDA_-_Grafana.png
|
|
| Attachment 12: Screenshot_2024-04-28_at_10-04-04_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 13: Screenshot_2024-04-28_at_10-04-19_ucesb.png
|
|
| Attachment 14: Screenshot_from_2024-04-28_10-08-23.png
|
|
| Attachment 15: Screenshot_from_2024-04-28_10-30-24.png
|
|
| Attachment 16: Screenshot_2024-04-28_at_10-34-47_AIDA_-_Grafana.png
|
|
| Attachment 17: Screenshot_2024-04-28_at_10-34-56_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 18: Capture_Grafana.PNG
|
|
| Attachment 19: Capture_Grafana_2.PNG
|
|
| Attachment 20: Screenshot_from_2024-04-28_10-58-43.png
|
|
| Attachment 21: Capture_Grafana_3.PNG
|
|
| Attachment 22: Screenshot_from_2024-04-28_12-00-49.png
|
|
| Attachment 23: Screenshot_2024-04-28_at_12-01-21_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 24: Screenshot_2024-04-28_at_12-02-58_Statistics_aidas-gsi.png
|
|
| Attachment 25: Screenshot_2024-04-28_at_12-02-36_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 26: Screenshot_2024-04-28_at_11-59-21_AIDA_-_Grafana.png
|
|
| Attachment 27: Screenshot_2024-04-28_at_11-59-44_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 28: Screenshot_2024-04-28_at_12-00-08_ucesb.png
|
|
| Attachment 29: Screenshot_2024-04-28_at_14-03-06_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 30: Screenshot_2024-04-28_at_14-04-21_Statistics_aidas-gsi.png
|
|
| Attachment 31: Screenshot_2024-04-28_at_14-04-07_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 32: Screenshot_2024-04-28_at_14-01-38_AIDA_-_Grafana.png
|
|
| Attachment 33: Screenshot_2024-04-28_at_14-01-52_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 34: Screenshot_2024-04-28_at_14-02-02_ucesb.png
|
|
| Attachment 35: Screenshot_from_2024-04-28_14-02-43.png
|
|
| Attachment 36: R21_500.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 259329603 ( 2382071.3 Hz)
Other data format: 2590397 ( 23794.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1783 ( 16.4 Hz)
RESUME: 1783 ( 16.4 Hz)
SYNC100: 32405 ( 297.7 Hz)
WR48-63: 32405 ( 297.7 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 2522021 ( 23166.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 289080 ( 2655.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 108.867 s
FEE elapsed dead time(s) elapsed idle time(s)
0 1.124 0.000
1 7.512 0.000
2 0.941 0.000
3 17.003 0.000
4 0.495 0.000
5 6.767 0.000
6 0.004 0.000
7 7.595 0.000
8 0.000 0.000
9 0.006 0.000
10 24.536 0.000
11 0.145 0.000
12 0.001 0.000
13 0.000 0.000
14 0.322 0.000
15 8.584 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 22169110 8971 0 0 57 57 2822 2822 0 3213 0 16255
1 12639517 112315 0 0 99 99 1612 1612 0 108893 0 29028
2 22281064 1127033 0 0 37 37 2985 2985 0 1120989 0 11197
3 20798124 911187 0 0 135 135 2715 2715 0 905487 0 8456
4 9363935 176936 0 0 14 14 1205 1205 0 174498 0 11991
5 27019969 216459 0 0 325 325 3434 3434 0 208941 0 9910
6 6076361 1558 0 0 2 2 777 777 0 0 0 3470
7 35565405 8998 0 0 189 189 4310 4310 0 0 0 153479
8 1477425 362 0 0 0 0 181 181 0 0 0 6096
9 6250272 1506 0 0 2 2 751 751 0 0 0 2572
10 27290515 8024 0 0 662 662 3350 3350 0 0 0 840
11 10865846 2620 0 0 16 16 1294 1294 0 0 0 22001
12 3883231 948 0 0 2 2 472 472 0 0 0 3485
13 3455565 826 0 0 0 0 413 413 0 0 0 4071
14 11216035 2794 0 0 51 51 1346 1346 0 0 0 2348
15 38977229 9860 0 0 192 192 4738 4738 0 0 0 3881
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 38.605s ( 828.898 blocks/s, 51.806 Mb/s)
|
|
614
|
Sat Apr 27 23:04:42 2024 |
Marc | 00:00-08:00 Sunday 28 April |
Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C
0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.
After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.
I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.
I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.
We are now back in business with AIDA back in main DAQ at 01:40
01:45 full checks:
Bias: AIDA-Graphana - Attachment 1
Leakage current ramping up again ! S4 temperaturenot really going down either: 25C
FEE64 temperatures OK - attachment 2
ADC data item stats - attachment 3
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 5
ucesb - attachment 6
03:00 full checks
No change to be highlighted. All pretty much the same as during previous full checks.
05:00 full checks
Expt still running very smoothly.
Bias: AIDA-Graphana - Attachment 7-8
Leakage current going down slightly
FEE64 temperatures OK - attachment 9
ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
per FEE64 Rate spectra - attachment 11
Merger etc - attachment 12
ucesb - attachment 13
07:15 all running smoothly.
07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.
07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.
|
| Attachment 1: Screenshot_from_2024-04-28_01-51-37.png
|
|
| Attachment 2: Screenshot_from_2024-04-28_01-53-29.png
|
|
| Attachment 3: Screenshot_from_2024-04-28_01-53-52.png
|
|
| Attachment 4: Screenshot_from_2024-04-28_01-54-16.png
|
|
| Attachment 5: Screenshot_from_2024-04-28_01-56-14.png
|
|
| Attachment 6: Screenshot_from_2024-04-28_04-59-46.png
|
|
| Attachment 7: Screenshot_from_2024-04-28_04-59-51.png
|
|
| Attachment 8: Screenshot_from_2024-04-28_05-00-33.png
|
|
| Attachment 9: Screenshot_from_2024-04-28_05-00-25.png
|
|
| Attachment 10: Screenshot_from_2024-04-28_05-00-38.png
|
|
| Attachment 11: Screenshot_from_2024-04-28_05-02-44.png
|
|
| Attachment 12: Screenshot_from_2024-04-28_05-00-03.png
|
|
|
613
|
Sat Apr 27 15:13:06 2024 |
Norah | 16:00 - 00:00 Saturday 27 April |
14:50 Checks
Leakage current is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1
It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.
18:15 Checks
Everything appears to be going smoothly
Grafana - DSSSD bias and leakage current - attachments 2 and 3
FEE64 temperatures - It appears okay, nothing strange , attachment 4
ADC data item stats - attachment 5
Per FEE64 Rate spectra - attachment 6
ucesb - attachment 7
Merger Link Data Rates - attachment 8
18:16
The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.
18:20 it came back to work normally
............................
20:00 Checks
Nothing new to report. All is well.
20:33 Aida02 showed as not reading - attachment 10
20:34 it resumed reading and returned to normal operation.
21:08 Aida02 was not reading - attachment 11 , had to emailed Tom to fix it .
The shift crew have restarted ucesb and AIDA is now showing OK - attachment 12
so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.
ADC data item stats and Merger etc appear to be going smoothly - attachments 13 and 14
...........................................
22:00 Checks
Still running smoothly.
DSSD bias and leakage current - ok
FEE64 temperatures - ok
Per FEE64 Rate spectra - ok
ADC data item stats - ok
Ucesb - ok
.................................
23:00 Checks
Nothing new to report. Everything is going well , and I have captured screenshots of each one
Leakage currents dropped a bit See attachment 15
.............................
23:50 Checks
All seems smooth. |
| Attachment 1: Screenshot_from_2024-04-27_16-54-13.png
|
|
| Attachment 2: Screenshot_from_2024-04-27_18-15-47.png
|
|
| Attachment 3: Screenshot_from_2024-04-27_18-09-13.png
|
|
| Attachment 4: Screenshot_from_2024-04-27_18-07-50.png
|
|
| Attachment 5: Screenshot_from_2024-04-27_18-06-46.png
|
|
| Attachment 6: Screenshot_from_2024-04-27_18-07-19.png
|
|
| Attachment 7: Screenshot_from_2024-04-27_18-08-53.png
|
|
| Attachment 8: Screenshot_from_2024-04-27_18-09-57.png
|
|
| Attachment 9: Screenshot_from_2024-04-27_19-18-03.png
|
|
| Attachment 10: Screenshot_from_2024-04-27_20-33-01.png
|
|
| Attachment 11: Screenshot_from_2024-04-27_20-52-28.png
|
|
| Attachment 12: Screenshot_from_2024-04-27_21-22-51.png
|
|
| Attachment 13: Screenshot_from_2024-04-27_21-44-02.png
|
|
| Attachment 14: Screenshot_from_2024-04-27_21-45-02.png
|
|
| Attachment 15: Screenshot_from_2024-04-27_23-26-06.png
|
|
|
612
|
Sat Apr 27 04:59:31 2024 |
TD | 08:00-16:00 Saturday 27 April |
06.00 DAQ continues file S100_alpha/R20_740
FRS setting 162Eu
DSSSD bias & leakage current - Grafana - attachments 1-2
leakage currents tracking S4 ambient temperature
FEE64 temperatures OK - attachment 3
All system wide checks OK *except*
as reported - attachments 4-5
*all* ADCs not calibrated (not calibrated post-synchronise ASIC clocks at most recent restart)
WR timestamps OK - attachment 6
ADC, PAUSE, RESUME, Correlation scaler data item stats OK - attachments 7-10
per FEE64 Rate spectra - attachment 11
per p+n FEE64 1.8.L specra - attachments 12-13
aida09 pulser peak width 55 ch FWHM (~38keV FWHM)
per FEE64 1.8.H spectra - attachments 14-15
Merger, TapeServer etc - attachments 16-17
12Mb/s to disk
ucesb - attachment 18
spill length 3.5s, extraction 1.5s
06.30 analysis data file R20_748 - attachment 19
max deadtime 12%
no timewarps
HEC data item rate 2.1kHz
06.35 new run file S100_alpha/R21
09.30 DAQ continues - file S100_alpha/R21_83
DSSSD bias & leakage currents - Grafana - attachment 20
FEE64 temperatures OK - attachment 21
ADC data item stats OK - attachments 22
per FEE64 Rate spectra - attachment 23
Merger etc - attachments 24
ucesb - attachment 25
spill length 3.5s, extraction 1.5s
09.50 per FEE64 1.8.W spectra - 20us FSR - attachments 26-27
no significant change in electronic noise cf. start of run
per p+n FEE64 1.8.L spectra - attachment 28
aida09 pulser peak width 62 ch FWHM
common x/y scale - peak heoght proxy for width
13.57 DAQ continues file R21_141
DSSSD bias & leakage current - Grafana - attachments 29-30
FEE64 temperatures OK - attachment 31
ADC data item stats OK - attachments 32
per p+n FEE64 1.8.L spectra - attachment 33
aida09 pulser peak width 55 ch FWHM (~39keV FWHM)
per FEE64 Rate spectra - attachment 34
Merger etc - attachments 35
ucesb - attachment 36
spill length 3.5s, extraction 1.5s
14.15 analysis data file R21_144 - attachment 35
max deadtime 16% (aida04)
no timewarps
HEC data item rate 2.0kHz |
| Attachment 1: Screenshot_from_2024-04-27_06-00-39.png
|
|
| Attachment 2: Screenshot_from_2024-04-27_06-01-02.png
|
|
| Attachment 3: Screenshot_from_2024-04-27_06-01-38.png
|
|
| Attachment 4: Screenshot_from_2024-04-27_06-02-22.png
|
|
| Attachment 5: Screenshot_from_2024-04-27_06-02-46.png
|
|
| Attachment 6: Screenshot_from_2024-04-27_06-03-00.png
|
|
| Attachment 7: Screenshot_from_2024-04-27_06-03-32.png
|
|
| Attachment 8: Screenshot_from_2024-04-27_06-04-34.png
|
|
| Attachment 9: Screenshot_from_2024-04-27_06-05-03.png
|
|
| Attachment 10: Screenshot_from_2024-04-27_06-05-37.png
|
|
| Attachment 11: Screenshot_from_2024-04-27_06-06-17.png
|
|
| Attachment 12: Screenshot_from_2024-04-27_06-07-30.png
|
|
| Attachment 13: Screenshot_from_2024-04-27_06-08-12.png
|
|
| Attachment 14: Screenshot_from_2024-04-27_06-11-26.png
|
|
| Attachment 15: Screenshot_from_2024-04-27_06-12-32.png
|
|
| Attachment 16: Screenshot_from_2024-04-27_06-13-07.png
|
|
| Attachment 17: Screenshot_from_2024-04-27_06-13-53.png
|
|
| Attachment 18: Screenshot_from_2024-04-27_06-14-37.png
|
|
| Attachment 19: R20_748
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 257504742 ( 1406851.0 Hz)
Other data format: 4415258 ( 24122.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1610 ( 8.8 Hz)
RESUME: 1611 ( 8.8 Hz)
SYNC100: 32682 ( 178.6 Hz)
WR48-63: 32682 ( 178.6 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 4346673 ( 23747.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 386233 ( 2110.1 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 183.036 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.284 0.000
1 10.840 0.000
2 0.188 0.000
3 23.217 0.000
4 0.100 0.000
5 2.070 0.000
6 0.000 0.000
7 2.151 0.000
8 0.000 0.000
9 0.000 0.000
10 8.754 0.000
11 0.009 0.000
12 0.000 0.000
13 0.000 0.000
14 0.015 0.000
15 2.919 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 12463516 8663 0 0 35 35 1556 1556 0 5481 0 27718
1 18658776 198614 0 0 164 164 2390 2390 0 193506 0 51705
2 15562218 1904264 0 0 13 13 2088 2088 0 1900062 0 19441
3 28663274 1575353 0 0 203 204 3714 3714 0 1567518 0 14524
4 7649540 297936 0 0 9 9 969 969 0 295980 0 20940
5 22941092 390586 0 0 220 220 3010 3010 0 384126 0 18537
6 5869094 1360 0 0 0 0 680 680 0 0 0 5932
7 40428040 10412 0 0 122 122 5084 5084 0 0 0 148981
8 2103382 490 0 0 0 0 245 245 0 0 0 10601
9 5530471 1492 0 0 0 0 746 746 0 0 0 4423
10 31486867 9242 0 0 711 711 3910 3910 0 0 0 1813
11 6680681 1722 0 0 2 2 859 859 0 0 0 37346
12 4842246 1236 0 0 0 0 618 618 0 0 0 5812
13 3504495 918 0 0 0 0 459 459 0 0 0 6910
14 6454647 1674 0 0 3 3 834 834 0 0 0 4246
15 44666403 11296 0 0 128 128 5520 5520 0 0 0 7304
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 38.639s ( 828.186 blocks/s, 51.762 Mb/s)
|
| Attachment 20: Capture.PNG
|
|
| Attachment 21: Screenshot_from_2024-04-27_09-32-29.png
|
|
| Attachment 22: Screenshot_from_2024-04-27_09-33-31.png
|
|
| Attachment 23: Screenshot_from_2024-04-27_09-34-20.png
|
|
| Attachment 24: Screenshot_from_2024-04-27_09-34-58.png
|
|
| Attachment 25: Screenshot_from_2024-04-27_09-35-41.png
|
|
| Attachment 26: Screenshot_from_2024-04-27_09-47-09.png
|
|
| Attachment 27: Screenshot_from_2024-04-27_09-48-06.png
|
|
| Attachment 28: Screenshot_from_2024-04-27_09-52-52.png
|
|
| Attachment 29: Screenshot_from_2024-04-27_13-57-44.png
|
|
| Attachment 30: Screenshot_from_2024-04-27_13-58-01.png
|
|
| Attachment 31: Screenshot_from_2024-04-27_14-01-26.png
|
|
| Attachment 32: Screenshot_from_2024-04-27_13-58-55.png
|
|
| Attachment 33: Screenshot_from_2024-04-27_14-03-49.png
|
|
| Attachment 34: Screenshot_from_2024-04-27_14-04-30.png
|
|
| Attachment 35: Screenshot_from_2024-04-27_14-05-30.png
|
|
| Attachment 36: Screenshot_from_2024-04-27_14-06-42.png
|
|
| Attachment 37: R21_144
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 256251858 ( 1109842.9 Hz)
Other data format: 5668143 ( 24549.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1144 ( 5.0 Hz)
RESUME: 1144 ( 5.0 Hz)
SYNC100: 32860 ( 142.3 Hz)
WR48-63: 32860 ( 142.3 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 5600135 ( 24254.5 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 452307 ( 1959.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 230.890 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.168 0.000
1 16.872 0.000
2 0.114 0.000
3 36.051 0.000
4 0.888 0.000
5 2.369 0.000
6 0.000 0.000
7 1.443 0.000
8 0.006 0.000
9 0.000 0.000
10 1.304 0.000
11 0.023 0.000
12 0.000 0.000
13 0.000 0.000
14 0.033 0.000
15 1.208 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 7949000 9111 0 0 19 19 1076 1076 0 6921 0 39332
1 26808065 269780 0 0 235 235 3356 3356 0 262598 0 70729
2 11614466 2448126 0 0 13 13 1817 1817 0 2444466 0 27115
3 37929923 1952070 0 0 294 294 4896 4896 0 1941690 0 19677
4 8915163 414541 0 0 12 12 1163 1163 0 412191 0 27553
5 22729514 538305 0 0 114 114 2904 2904 0 532269 0 25872
6 5822184 1488 0 0 0 0 744 744 0 0 0 8358
7 40443466 10506 0 0 99 99 5154 5154 0 0 0 123022
8 3043811 744 0 0 1 1 371 371 0 0 0 14593
9 5157550 1312 0 0 0 0 656 656 0 0 0 6148
10 26044004 6988 0 0 273 273 3221 3221 0 0 0 2629
11 5888316 1514 0 0 4 4 753 753 0 0 0 52905
12 5831030 1426 0 0 0 0 713 713 0 0 0 8360
13 4114996 986 0 0 0 0 493 493 0 0 0 9929
14 5361222 1380 0 0 5 5 685 685 0 0 0 5844
15 38599148 9866 0 0 75 75 4858 4858 0 0 0 10241
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 33.879s ( 944.541 blocks/s, 59.034 Mb/s)
|
|
611
|
Fri Apr 26 23:04:51 2024 |
Marc | 00:00-08:00 Saturday 27 April |
It's a new day, it's a new shift and I'm feeeeeliiiiing goooood !!!!!
0.15 Checks:
DSSSD bias & leakage current - Grafana - attachments 1-2
leakage current dropped a bit since the evening shift ~22:00.
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 4
per FEE64 Rate spectra - attachment 5
Merger etc - attachment 6
ucesb - attachment 7
2:20 Nothing new to report. All good. Leakage currents continue to go down slowly. See attachement 8 to 12
3:30 Still running smoothly.
5:20 AIDA 4 stop counting in the last 20 min. I informed the main DAQ and they started a new run without AIDA.
DAQ restarted OK. Main DAQ informed.
|
| Attachment 1: Screenshot_from_2024-04-27_00-17-00.png
|
|
| Attachment 2: Screenshot_from_2024-04-27_00-16-16.png
|
|
| Attachment 3: Screenshot_from_2024-04-27_00-18-49.png
|
|
| Attachment 4: Screenshot_from_2024-04-27_00-19-05.png
|
|
| Attachment 5: Screenshot_from_2024-04-27_00-19-19.png
|
|
| Attachment 6: Screenshot_from_2024-04-27_00-21-44.png
|
|
| Attachment 7: Screenshot_from_2024-04-27_00-21-00.png
|
|
| Attachment 8: Screenshot_from_2024-04-27_02-32-22.png
|
|
| Attachment 9: Screenshot_from_2024-04-27_02-37-20.png
|
|
| Attachment 10: Screenshot_from_2024-04-27_02-37-31.png
|
|
| Attachment 11: Screenshot_from_2024-04-27_02-40-39.png
|
|
| Attachment 12: Screenshot_from_2024-04-27_02-41-13.png
|
|
|
610
|
Fri Apr 26 15:34:00 2024 |
RDP | 16:00 - 00:00 Friday 26 April |
This appeared in the system log:
Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors
These messages seem to appear sporadically.
18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?
18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.
aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters
Thanks Nic!
22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)
22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.
|
| Attachment 1: Screenshot_2024-04-26_at_16-40-03_ucesb.png
|
|
| Attachment 2: Screenshot_2024-04-26_at_16-41-57_AIDA_-_Grafana.png
|
|
| Attachment 3: Screenshot_2024-04-26_at_19-07-21_Merger_Input_Data_Link_Rates_aidas-gsi.png
|
|
| Attachment 4: Screenshot_2024-04-26_at_19-18-28_ucesb.png
|
|
| Attachment 5: Screenshot_2024-04-26_at_22-12-06_ucesb.png
|
|
| Attachment 6: Screenshot_2024-04-26_at_22-11-47_AIDA_-_Grafana.png
|
|
|
609
|
Fri Apr 26 08:15:59 2024 |
TD | [How To] Recover DAQ when one FEE64 stops sending data |
Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running *and* data is being transferred to the Merger and TapeServer.
Typical symptoms are:
- no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser']
- zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics']
- Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control']
active data links are green, inactive links are red
N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ
1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'
2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
terminal window showing the startup will popup
3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']
4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
all data links will be inactive (red)
5) Reboot problem FEE64s
Launch a command terminal and telnet to the problem FEE64 for example
telnet aida04
Enter user 'root' and enter password. When logged in reboot the FEE64 using the command
reboot
6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal']
you should see two messages showing the FEE64 mounting filesystems
wait c. three minutes for boot to complete
7) Restart DAQ
Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control']
Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control']
Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control']
Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control']
Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control']
Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']
8) Restore ASIC settings
Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control']
Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings']
Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings']
Wait 10s
9) Check ASIC settings
Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control']
Wait for popup indicating completion of checks and dismiss
10) System wide checks
Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check clock status' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks']
make a note or take a screenshot of any error messages observed
11) Start DAQ
Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control']
Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
readout times will be displayed for each FEE64
check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support
13) Stop DAQ
Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ has *actually* stopped - if not repeat 'STOP
14) Enable data transfer to Merger and TapeServer and start DAQ
Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
15) Check data is being transferred to Merger and TapeServer
Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)
Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
you should see 'Blocks written' and 'kb written' counters increase
16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]
17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.
If not, please request expert assistance.
|
|
608
|
Fri Apr 26 07:01:26 2024 |
Norah | 08:00 - 16:00 Friday 26 April |
10:00 Checks:
All seems smooth.
DSSD bias and leakage current ( Attachment 1)
FEE64 temperatures (Attachment 2)
Per FEE64 Rate spectra (Attachment 3)
ADC data item stats -(Attachment 4)
Ucesb (Attachment 5)
10:21
Aida05 was zero not reading had to call Nic to fix it.
10:34
Now, Aida05 is running well.
.......................................................
12:00 Checks:
Everything appears to be going smoothly, and I took screenshot of them.
12:28, Aida02 showed as not reading , had to email Nic to fix it.
12:30, it resumed reading and returned to normal operation.
.........................................................
14:00 Checks:
Everything is going well , and I have captured screenshots of each one.
......................................................
16:00 Checks:
All seems smooth, and I took screenshot of them.
|
| Attachment 1: Screenshot_from_2024-04-26_10-00-39.png
|
|
| Attachment 2: Screenshot_from_2024-04-26_10-02-35.png
|
|
| Attachment 3: Screenshot_from_2024-04-26_10-04-40.png
|
|
| Attachment 4: Screenshot_from_2024-04-26_10-06-18.png
|
|
| Attachment 5: Screenshot_from_2024-04-26_10-09-52.png
|
|
|
607
|
Thu Apr 25 22:45:30 2024 |
Tom, Muneerah, Norah | Over night shift AIDA 26 Fri |
01:02
Aida04 was zero not reading had to call Tom to fix it.
02:00 checked all good and attached.
Its been running good after that till 7:30 again the same problem happened and Tom fixed it.
|
| Attachment 1: Screenshot_from_2024-04-25_23-42-15.png
|
|
| Attachment 2: Screenshot_from_2024-04-26_00-05-47.png
|
|
| Attachment 3: Screenshot_from_2024-04-26_00-05-15.png
|
|
| Attachment 4: Screenshot_from_2024-04-26_00-04-22.png
|
|
| Attachment 5: Screenshot_from_2024-04-26_02-00-17.png
|
|
| Attachment 6: Screenshot_from_2024-04-26_01-58-39.png
|
|
| Attachment 7: Screenshot_from_2024-04-26_01-58-39.png
|
|
| Attachment 8: Screenshot_from_2024-04-26_01-57-38.png
|
|
| Attachment 9: Screenshot_from_2024-04-26_01-57-01.png
|
|