| |
ID |
Date |
Author |
Subject |
|
|
621
|
Sun Apr 28 22:52:06 2024 |
Marc | 0:00-08:00 Monday 29 April |
Just another manic magic Monday, ....
Taking over Betool's AIDA shift. All is good.
This shift's first hour has been eventless. Temperatures, Rates, bias, all look good.
01:15: AIDA04 is down again. DAQ shift crew informed.
01:27: AIDA04 is back and shift crew has been informed.
01:30 Checks:
Bias and leakage current - Attachment 1-2
Temperatures - Attachment 3
ADC Data item - Attachment 4
Rate Spectra - Attachment 5
UCESB - Attachment 6
Timestamp checks - Attachement 7
06:00 It has run very smoothly.
|
| Attachment 1: Screenshot_from_2024-04-29_01-33-27.png
|
|
| Attachment 2: Screenshot_from_2024-04-29_01-34-37.png
|
|
| Attachment 3: Screenshot_from_2024-04-29_01-36-12.png
|
|
| Attachment 4: Screenshot_from_2024-04-29_01-36-41.png
|
|
| Attachment 5: Screenshot_from_2024-04-29_01-37-56.png
|
|
| Attachment 6: Screenshot_from_2024-04-29_01-32-39.png
|
|
| Attachment 7: Screenshot_from_2024-04-29_01-42-58.png
|
|
|
|
620
|
Sun Apr 28 20:35:57 2024 |
Betool Alayed | 16:0-00:00 28 April 2024 |
21:30 screenshots
and
23:24 screenshots |
| Attachment 1: Screenshot_from_2024-04-28_21-30-19.png
|
|
| Attachment 2: Screenshot_from_2024-04-28_21-31-11.png
|
|
| Attachment 3: Screenshot_from_2024-04-28_21-32-07.png
|
|
| Attachment 4: Screenshot_from_2024-04-28_21-32-51.png
|
|
| Attachment 5: Screenshot_from_2024-04-28_21-35-10.png
|
|
| Attachment 6: Screenshot_from_2024-04-28_23-35-25.png
|
|
| Attachment 7: Screenshot_from_2024-04-28_23-34-31.png
|
|
| Attachment 8: Screenshot_from_2024-04-28_23-33-42.png
|
|
| Attachment 9: Screenshot_from_2024-04-28_23-32-28.png
|
|
| Attachment 10: Screenshot_from_2024-04-28_23-31-51.png
|
|
|
|
619
|
Sun Apr 28 16:26:47 2024 |
Betool Alayed | 16:00-00:00 Sunday 28 April |
19:24 pm screenshots
and
17:18 pm screenshots |
| Attachment 1: Screenshot_from_2024-04-28_17-18-38.png
|
|
| Attachment 2: Screenshot_from_2024-04-28_17-13-13.png
|
|
| Attachment 3: Screenshot_from_2024-04-28_17-11-48.png
|
|
| Attachment 4: Screenshot_from_2024-04-28_17-10-32.png
|
|
| Attachment 5: Screenshot_from_2024-04-28_17-04-10.png
|
|
| Attachment 6: Screenshot_from_2024-04-28_19-18-42.png
|
|
| Attachment 7: Screenshot_from_2024-04-28_19-20-10.png
|
|
| Attachment 8: Screenshot_from_2024-04-28_19-22-19.png
|
|
| Attachment 9: Screenshot_from_2024-04-28_19-23-19.png
|
|
| Attachment 10: Screenshot_from_2024-04-28_19-24-40.png
|
|
|
|
618
|
Sun Apr 28 16:07:59 2024 |
TD | [How To] Recover DAQ when one FEE64 stops sending data - short(er) version |
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ
1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'
2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
terminal window showing the startup will popup
3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']
4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
all data links will be inactive (red)
5) Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']
6) Start DAQ
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
7) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
readout times will be displayed for each FEE64
check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support
8) Stop DAQ
Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ has *actually* stopped - if not repeat 'STOP
9) Enable data transfer to Merger and TapeServer and start DAQ
Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
10) Check data is being transferred to Merger and TapeServer
Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)
Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
you should see 'Blocks written' and 'kb written' counters increase
11) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]
12) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.
If not, please request expert assistance. |
|
|
617
|
Sun Apr 28 15:49:58 2024 |
TD | DAQ restart short(er) cut |
16.45 aida04 stopped producing data
DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
aida04 did not stop, all other FEE64s stopped
restart Merger (workspace 3, display 2, select 'NewMerger' icon)
SETUP Merger (workspace 3, display 1, browser 'NewMerger' tab)
GO Merger (workspace 3, display 1, browser 'NewMerger' tab)
aida04 now stops
disable *all* #1 data transfer
DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
check WR timestamps OK (workspace 2, display 1, GSI WR Timestamp Control)
DAQ STOP (workspace 2, display 1, browser 'Run Control' tab)
enable *all* #1 data transfer
DAQ GO (workspace 2, display 1, browser 'Run Control' tab)
No reboot or power cycle required. Just clear the Merger data links, restart DAQ.
|
|
|
616
|
Sun Apr 28 15:09:34 2024 |
Betool Alayed | 16:0-00:00 28 April 2024 |
16:00 aida04 not producing data
power cycle all FEE64s to recover DAQ
16.20 analysis data file R21_668
max deadtime 17% (aida04), 9% (aida02), 2% (aida06) all others < 1%
no timewarps
HEC data item rate 1.9kHz
|
| Attachment 1: R21_668.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 256513573 ( 1160512.7 Hz)
Other data format: 5406427 ( 24459.6 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1079 ( 4.9 Hz)
RESUME: 1078 ( 4.9 Hz)
SYNC100: 32824 ( 148.5 Hz)
WR48-63: 32824 ( 148.5 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 5338622 ( 24152.9 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 440937 ( 1994.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 221.035 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.085 0.000
1 19.736 0.000
2 0.053 0.000
3 36.607 0.000
4 0.854 0.000
5 4.747 0.000
6 0.000 0.000
7 1.845 0.000
8 0.000 0.000
9 0.000 0.000
10 0.661 0.000
11 0.020 0.000
12 0.004 0.000
13 0.000 0.000
14 0.003 0.000
15 1.247 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 8138410 8624 0 0 12 12 985 985 0 6630 0 38979
1 27142829 245404 0 0 242 242 3461 3461 0 237998 0 64179
2 11096772 2350080 0 0 8 8 1768 1768 0 2346528 0 26862
3 37888832 1835283 0 0 304 303 4762 4762 0 1825152 0 18340
4 9329455 413600 0 0 28 28 1212 1212 0 411120 0 29227
5 23846951 517930 0 0 167 167 3201 3201 0 511194 0 24385
6 5635241 1420 0 0 0 0 710 710 0 0 0 8053
7 40074524 10318 0 0 93 93 5066 5066 0 0 0 120554
8 3088839 754 0 0 0 0 377 377 0 0 0 14672
9 5102920 1254 0 0 0 0 627 627 0 0 0 6080
10 25963054 6756 0 0 140 140 3238 3238 0 0 0 2640
11 5980678 1458 0 0 5 5 724 724 0 0 0 52714
12 5769255 1548 0 0 2 2 772 772 0 0 0 8398
13 3930590 926 0 0 0 0 463 463 0 0 0 9544
14 5399163 1328 0 0 1 1 663 663 0 0 0 5692
15 38126060 9744 0 0 77 77 4795 4795 0 0 0 10618
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.004s ( 941.068 blocks/s, 58.817 Mb/s)
|
|
|
615
|
Sun Apr 28 06:57:59 2024 |
Magda Satrazani | 08:00-16:00 Sunday 28 April |
Took over the shift from Marc.
10.51 Added screenshots to illustrate erroneous data from Grafana (AIDA) - attachments 18-19
Compare to the CAEN HV control terminal (attachment 15) which shows the expected values. Have email'd Nic & Helena. TD
10.55 Noticed that the CAEN HV control terminal is showing the corerct data but on the wrong line of the display (attachment 15). Enter commands q (quit) and d (display) to restart display - the correct data now appears on the correct line (attachment 20).
Grafana (AIDA) is now reporting data correctly too (attachment 21).
Summary - CAEN HV terminal display issue ( c. 10.05 - 10.55 )
Note the correct data/values continued to be shown. Leakage current shown correctly by both Grafana and CAEN HV control throughout.
Conclude this was a display issue - not a change in the operating parameters of the CAEN N1419ET.. Data should be OK.
14.45 analysis data file R21_500 - attachment 36
max deadtime 22% (aida11), 16% (aida04), 7% (aida02), 6% (aida08), 6% (aida06) , all others < 1%
no timewarps
HEC data item rate 2.7kHz
|
| Attachment 1: Screenshot_2024-04-28_at_08-00-09_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 2: Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
|
|
| Attachment 3: leakage_current.png
|
|
| Attachment 4: Screenshot_2024-04-28_at_08-05-38_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 5: Screenshot_2024-04-28_at_08-08-14_ucesb.png
|
|
| Attachment 6: Screenshot_2024-04-28_at_08-09-21_AIDA_-_Grafana.png
|
|
| Attachment 7: Screenshot_2024-04-28_at_08-11-20_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 8: Screenshot_2024-04-28_at_10-01-48_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 9: Screenshot_2024-04-28_at_08-01-18_Statistics_aidas-gsi.png
|
|
| Attachment 10: Screenshot_2024-04-28_at_10-02-44_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 11: Screenshot_2024-04-28_at_10-03-40_AIDA_-_Grafana.png
|
|
| Attachment 12: Screenshot_2024-04-28_at_10-04-04_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 13: Screenshot_2024-04-28_at_10-04-19_ucesb.png
|
|
| Attachment 14: Screenshot_from_2024-04-28_10-08-23.png
|
|
| Attachment 15: Screenshot_from_2024-04-28_10-30-24.png
|
|
| Attachment 16: Screenshot_2024-04-28_at_10-34-47_AIDA_-_Grafana.png
|
|
| Attachment 17: Screenshot_2024-04-28_at_10-34-56_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 18: Capture_Grafana.PNG
|
|
| Attachment 19: Capture_Grafana_2.PNG
|
|
| Attachment 20: Screenshot_from_2024-04-28_10-58-43.png
|
|
| Attachment 21: Capture_Grafana_3.PNG
|
|
| Attachment 22: Screenshot_from_2024-04-28_12-00-49.png
|
|
| Attachment 23: Screenshot_2024-04-28_at_12-01-21_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 24: Screenshot_2024-04-28_at_12-02-58_Statistics_aidas-gsi.png
|
|
| Attachment 25: Screenshot_2024-04-28_at_12-02-36_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 26: Screenshot_2024-04-28_at_11-59-21_AIDA_-_Grafana.png
|
|
| Attachment 27: Screenshot_2024-04-28_at_11-59-44_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 28: Screenshot_2024-04-28_at_12-00-08_ucesb.png
|
|
| Attachment 29: Screenshot_2024-04-28_at_14-03-06_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 30: Screenshot_2024-04-28_at_14-04-21_Statistics_aidas-gsi.png
|
|
| Attachment 31: Screenshot_2024-04-28_at_14-04-07_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 32: Screenshot_2024-04-28_at_14-01-38_AIDA_-_Grafana.png
|
|
| Attachment 33: Screenshot_2024-04-28_at_14-01-52_AIDA_Alerting_-_Grafana.png
|
|
| Attachment 34: Screenshot_2024-04-28_at_14-02-02_ucesb.png
|
|
| Attachment 35: Screenshot_from_2024-04-28_14-02-43.png
|
|
| Attachment 36: R21_500.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 259329603 ( 2382071.3 Hz)
Other data format: 2590397 ( 23794.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1783 ( 16.4 Hz)
RESUME: 1783 ( 16.4 Hz)
SYNC100: 32405 ( 297.7 Hz)
WR48-63: 32405 ( 297.7 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 2522021 ( 23166.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 289080 ( 2655.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 108.867 s
FEE elapsed dead time(s) elapsed idle time(s)
0 1.124 0.000
1 7.512 0.000
2 0.941 0.000
3 17.003 0.000
4 0.495 0.000
5 6.767 0.000
6 0.004 0.000
7 7.595 0.000
8 0.000 0.000
9 0.006 0.000
10 24.536 0.000
11 0.145 0.000
12 0.001 0.000
13 0.000 0.000
14 0.322 0.000
15 8.584 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 22169110 8971 0 0 57 57 2822 2822 0 3213 0 16255
1 12639517 112315 0 0 99 99 1612 1612 0 108893 0 29028
2 22281064 1127033 0 0 37 37 2985 2985 0 1120989 0 11197
3 20798124 911187 0 0 135 135 2715 2715 0 905487 0 8456
4 9363935 176936 0 0 14 14 1205 1205 0 174498 0 11991
5 27019969 216459 0 0 325 325 3434 3434 0 208941 0 9910
6 6076361 1558 0 0 2 2 777 777 0 0 0 3470
7 35565405 8998 0 0 189 189 4310 4310 0 0 0 153479
8 1477425 362 0 0 0 0 181 181 0 0 0 6096
9 6250272 1506 0 0 2 2 751 751 0 0 0 2572
10 27290515 8024 0 0 662 662 3350 3350 0 0 0 840
11 10865846 2620 0 0 16 16 1294 1294 0 0 0 22001
12 3883231 948 0 0 2 2 472 472 0 0 0 3485
13 3455565 826 0 0 0 0 413 413 0 0 0 4071
14 11216035 2794 0 0 51 51 1346 1346 0 0 0 2348
15 38977229 9860 0 0 192 192 4738 4738 0 0 0 3881
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 38.605s ( 828.898 blocks/s, 51.806 Mb/s)
|
|
|
614
|
Sat Apr 27 23:04:42 2024 |
Marc | 00:00-08:00 Sunday 28 April |
Encore un matin, un matin pour ..... a shift !!!ramping up again ! S4 temperature not really going down either: 25C
0:40 AIDA04 is down. I informed the DAQ shift crew and attempting to restart AIDA04.
After rebooting AIDA04 all seemed going until I reached the merger check. No data was being merged.
I realise then that AIDA01 data transfer was not active. I did not spot it when I enabled the data transfer for all.
I stopped the daq and rebooted AIDA01 following all the instructions in the ELOG and this worked.
We are now back in business with AIDA back in main DAQ at 01:40
01:45 full checks:
Bias: AIDA-Graphana - Attachment 1
Leakage current ramping up again ! S4 temperaturenot really going down either: 25C
FEE64 temperatures OK - attachment 2
ADC data item stats - attachment 3
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 5
ucesb - attachment 6
03:00 full checks
No change to be highlighted. All pretty much the same as during previous full checks.
05:00 full checks
Expt still running very smoothly.
Bias: AIDA-Graphana - Attachment 7-8
Leakage current going down slightly
FEE64 temperatures OK - attachment 9
ADC data item stats - attachment 10
ramping up again ! S4 temperaturenot really going down either: 25C
per FEE64 Rate spectra - attachment 11
Merger etc - attachment 12
ucesb - attachment 13
07:15 all running smoothly.
07:20 Spoke too fast AIDA DAQ is down. DAQ shift crew informed.
07:40: AIDA10 was rebooted successfully and AIDA is now back in the main DAQ.
|
| Attachment 1: Screenshot_from_2024-04-28_01-51-37.png
|
|
| Attachment 2: Screenshot_from_2024-04-28_01-53-29.png
|
|
| Attachment 3: Screenshot_from_2024-04-28_01-53-52.png
|
|
| Attachment 4: Screenshot_from_2024-04-28_01-54-16.png
|
|
| Attachment 5: Screenshot_from_2024-04-28_01-56-14.png
|
|
| Attachment 6: Screenshot_from_2024-04-28_04-59-46.png
|
|
| Attachment 7: Screenshot_from_2024-04-28_04-59-51.png
|
|
| Attachment 8: Screenshot_from_2024-04-28_05-00-33.png
|
|
| Attachment 9: Screenshot_from_2024-04-28_05-00-25.png
|
|
| Attachment 10: Screenshot_from_2024-04-28_05-00-38.png
|
|
| Attachment 11: Screenshot_from_2024-04-28_05-02-44.png
|
|
| Attachment 12: Screenshot_from_2024-04-28_05-00-03.png
|
|
|
|
613
|
Sat Apr 27 15:13:06 2024 |
Norah | 16:00 - 00:00 Saturday 27 April |
14:50 Checks
Leakage current is 20.021 and HV status is Max Voltages ! Is it normal ? - attachments 1
It is quite warm at GSI today and this may be a temperature effect. Hopefully will go down latter in the day (Marc). Any way let's kep an eye it. The current threshold is set to 30uA.
18:15 Checks
Everything appears to be going smoothly
Grafana - DSSSD bias and leakage current - attachments 2 and 3
FEE64 temperatures - It appears okay, nothing strange , attachment 4
ADC data item stats - attachment 5
Per FEE64 Rate spectra - attachment 6
ucesb - attachment 7
Merger Link Data Rates - attachment 8
18:16
The temperature for aida07 gave us No response - attachment 9 , I had emailed Tom to fix.
18:20 it came back to work normally
............................
20:00 Checks
Nothing new to report. All is well.
20:33 Aida02 showed as not reading - attachment 10
20:34 it resumed reading and returned to normal operation.
21:08 Aida02 was not reading - attachment 11 , had to emailed Tom to fix it .
The shift crew have restarted ucesb and AIDA is now showing OK - attachment 12
so Tom guess the problem really was at their end . They may still have a problem with the FRS DAQ.
ADC data item stats and Merger etc appear to be going smoothly - attachments 13 and 14
...........................................
22:00 Checks
Still running smoothly.
DSSD bias and leakage current - ok
FEE64 temperatures - ok
Per FEE64 Rate spectra - ok
ADC data item stats - ok
Ucesb - ok
.................................
23:00 Checks
Nothing new to report. Everything is going well , and I have captured screenshots of each one
Leakage currents dropped a bit See attachment 15
.............................
23:50 Checks
All seems smooth. |
| Attachment 1: Screenshot_from_2024-04-27_16-54-13.png
|
|
| Attachment 2: Screenshot_from_2024-04-27_18-15-47.png
|
|
| Attachment 3: Screenshot_from_2024-04-27_18-09-13.png
|
|
| Attachment 4: Screenshot_from_2024-04-27_18-07-50.png
|
|
| Attachment 5: Screenshot_from_2024-04-27_18-06-46.png
|
|
| Attachment 6: Screenshot_from_2024-04-27_18-07-19.png
|
|
| Attachment 7: Screenshot_from_2024-04-27_18-08-53.png
|
|
| Attachment 8: Screenshot_from_2024-04-27_18-09-57.png
|
|
| Attachment 9: Screenshot_from_2024-04-27_19-18-03.png
|
|
| Attachment 10: Screenshot_from_2024-04-27_20-33-01.png
|
|
| Attachment 11: Screenshot_from_2024-04-27_20-52-28.png
|
|
| Attachment 12: Screenshot_from_2024-04-27_21-22-51.png
|
|
| Attachment 13: Screenshot_from_2024-04-27_21-44-02.png
|
|
| Attachment 14: Screenshot_from_2024-04-27_21-45-02.png
|
|
| Attachment 15: Screenshot_from_2024-04-27_23-26-06.png
|
|
|
|
612
|
Sat Apr 27 04:59:31 2024 |
TD | 08:00-16:00 Saturday 27 April |
06.00 DAQ continues file S100_alpha/R20_740
FRS setting 162Eu
DSSSD bias & leakage current - Grafana - attachments 1-2
leakage currents tracking S4 ambient temperature
FEE64 temperatures OK - attachment 3
All system wide checks OK *except*
as reported - attachments 4-5
*all* ADCs not calibrated (not calibrated post-synchronise ASIC clocks at most recent restart)
WR timestamps OK - attachment 6
ADC, PAUSE, RESUME, Correlation scaler data item stats OK - attachments 7-10
per FEE64 Rate spectra - attachment 11
per p+n FEE64 1.8.L specra - attachments 12-13
aida09 pulser peak width 55 ch FWHM (~38keV FWHM)
per FEE64 1.8.H spectra - attachments 14-15
Merger, TapeServer etc - attachments 16-17
12Mb/s to disk
ucesb - attachment 18
spill length 3.5s, extraction 1.5s
06.30 analysis data file R20_748 - attachment 19
max deadtime 12%
no timewarps
HEC data item rate 2.1kHz
06.35 new run file S100_alpha/R21
09.30 DAQ continues - file S100_alpha/R21_83
DSSSD bias & leakage currents - Grafana - attachment 20
FEE64 temperatures OK - attachment 21
ADC data item stats OK - attachments 22
per FEE64 Rate spectra - attachment 23
Merger etc - attachments 24
ucesb - attachment 25
spill length 3.5s, extraction 1.5s
09.50 per FEE64 1.8.W spectra - 20us FSR - attachments 26-27
no significant change in electronic noise cf. start of run
per p+n FEE64 1.8.L spectra - attachment 28
aida09 pulser peak width 62 ch FWHM
common x/y scale - peak heoght proxy for width
13.57 DAQ continues file R21_141
DSSSD bias & leakage current - Grafana - attachments 29-30
FEE64 temperatures OK - attachment 31
ADC data item stats OK - attachments 32
per p+n FEE64 1.8.L spectra - attachment 33
aida09 pulser peak width 55 ch FWHM (~39keV FWHM)
per FEE64 Rate spectra - attachment 34
Merger etc - attachments 35
ucesb - attachment 36
spill length 3.5s, extraction 1.5s
14.15 analysis data file R21_144 - attachment 35
max deadtime 16% (aida04)
no timewarps
HEC data item rate 2.0kHz |
| Attachment 1: Screenshot_from_2024-04-27_06-00-39.png
|
|
| Attachment 2: Screenshot_from_2024-04-27_06-01-02.png
|
|
| Attachment 3: Screenshot_from_2024-04-27_06-01-38.png
|
|
| Attachment 4: Screenshot_from_2024-04-27_06-02-22.png
|
|
| Attachment 5: Screenshot_from_2024-04-27_06-02-46.png
|
|
| Attachment 6: Screenshot_from_2024-04-27_06-03-00.png
|
|
| Attachment 7: Screenshot_from_2024-04-27_06-03-32.png
|
|
| Attachment 8: Screenshot_from_2024-04-27_06-04-34.png
|
|
| Attachment 9: Screenshot_from_2024-04-27_06-05-03.png
|
|
| Attachment 10: Screenshot_from_2024-04-27_06-05-37.png
|
|
| Attachment 11: Screenshot_from_2024-04-27_06-06-17.png
|
|
| Attachment 12: Screenshot_from_2024-04-27_06-07-30.png
|
|
| Attachment 13: Screenshot_from_2024-04-27_06-08-12.png
|
|
| Attachment 14: Screenshot_from_2024-04-27_06-11-26.png
|
|
| Attachment 15: Screenshot_from_2024-04-27_06-12-32.png
|
|
| Attachment 16: Screenshot_from_2024-04-27_06-13-07.png
|
|
| Attachment 17: Screenshot_from_2024-04-27_06-13-53.png
|
|
| Attachment 18: Screenshot_from_2024-04-27_06-14-37.png
|
|
| Attachment 19: R20_748
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 257504742 ( 1406851.0 Hz)
Other data format: 4415258 ( 24122.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1610 ( 8.8 Hz)
RESUME: 1611 ( 8.8 Hz)
SYNC100: 32682 ( 178.6 Hz)
WR48-63: 32682 ( 178.6 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 4346673 ( 23747.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 386233 ( 2110.1 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 183.036 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.284 0.000
1 10.840 0.000
2 0.188 0.000
3 23.217 0.000
4 0.100 0.000
5 2.070 0.000
6 0.000 0.000
7 2.151 0.000
8 0.000 0.000
9 0.000 0.000
10 8.754 0.000
11 0.009 0.000
12 0.000 0.000
13 0.000 0.000
14 0.015 0.000
15 2.919 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 12463516 8663 0 0 35 35 1556 1556 0 5481 0 27718
1 18658776 198614 0 0 164 164 2390 2390 0 193506 0 51705
2 15562218 1904264 0 0 13 13 2088 2088 0 1900062 0 19441
3 28663274 1575353 0 0 203 204 3714 3714 0 1567518 0 14524
4 7649540 297936 0 0 9 9 969 969 0 295980 0 20940
5 22941092 390586 0 0 220 220 3010 3010 0 384126 0 18537
6 5869094 1360 0 0 0 0 680 680 0 0 0 5932
7 40428040 10412 0 0 122 122 5084 5084 0 0 0 148981
8 2103382 490 0 0 0 0 245 245 0 0 0 10601
9 5530471 1492 0 0 0 0 746 746 0 0 0 4423
10 31486867 9242 0 0 711 711 3910 3910 0 0 0 1813
11 6680681 1722 0 0 2 2 859 859 0 0 0 37346
12 4842246 1236 0 0 0 0 618 618 0 0 0 5812
13 3504495 918 0 0 0 0 459 459 0 0 0 6910
14 6454647 1674 0 0 3 3 834 834 0 0 0 4246
15 44666403 11296 0 0 128 128 5520 5520 0 0 0 7304
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 38.639s ( 828.186 blocks/s, 51.762 Mb/s)
|
| Attachment 20: Capture.PNG
|
|
| Attachment 21: Screenshot_from_2024-04-27_09-32-29.png
|
|
| Attachment 22: Screenshot_from_2024-04-27_09-33-31.png
|
|
| Attachment 23: Screenshot_from_2024-04-27_09-34-20.png
|
|
| Attachment 24: Screenshot_from_2024-04-27_09-34-58.png
|
|
| Attachment 25: Screenshot_from_2024-04-27_09-35-41.png
|
|
| Attachment 26: Screenshot_from_2024-04-27_09-47-09.png
|
|
| Attachment 27: Screenshot_from_2024-04-27_09-48-06.png
|
|
| Attachment 28: Screenshot_from_2024-04-27_09-52-52.png
|
|
| Attachment 29: Screenshot_from_2024-04-27_13-57-44.png
|
|
| Attachment 30: Screenshot_from_2024-04-27_13-58-01.png
|
|
| Attachment 31: Screenshot_from_2024-04-27_14-01-26.png
|
|
| Attachment 32: Screenshot_from_2024-04-27_13-58-55.png
|
|
| Attachment 33: Screenshot_from_2024-04-27_14-03-49.png
|
|
| Attachment 34: Screenshot_from_2024-04-27_14-04-30.png
|
|
| Attachment 35: Screenshot_from_2024-04-27_14-05-30.png
|
|
| Attachment 36: Screenshot_from_2024-04-27_14-06-42.png
|
|
| Attachment 37: R21_144
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 256251858 ( 1109842.9 Hz)
Other data format: 5668143 ( 24549.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1144 ( 5.0 Hz)
RESUME: 1144 ( 5.0 Hz)
SYNC100: 32860 ( 142.3 Hz)
WR48-63: 32860 ( 142.3 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 5600135 ( 24254.5 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 452307 ( 1959.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 230.890 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.168 0.000
1 16.872 0.000
2 0.114 0.000
3 36.051 0.000
4 0.888 0.000
5 2.369 0.000
6 0.000 0.000
7 1.443 0.000
8 0.006 0.000
9 0.000 0.000
10 1.304 0.000
11 0.023 0.000
12 0.000 0.000
13 0.000 0.000
14 0.033 0.000
15 1.208 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 7949000 9111 0 0 19 19 1076 1076 0 6921 0 39332
1 26808065 269780 0 0 235 235 3356 3356 0 262598 0 70729
2 11614466 2448126 0 0 13 13 1817 1817 0 2444466 0 27115
3 37929923 1952070 0 0 294 294 4896 4896 0 1941690 0 19677
4 8915163 414541 0 0 12 12 1163 1163 0 412191 0 27553
5 22729514 538305 0 0 114 114 2904 2904 0 532269 0 25872
6 5822184 1488 0 0 0 0 744 744 0 0 0 8358
7 40443466 10506 0 0 99 99 5154 5154 0 0 0 123022
8 3043811 744 0 0 1 1 371 371 0 0 0 14593
9 5157550 1312 0 0 0 0 656 656 0 0 0 6148
10 26044004 6988 0 0 273 273 3221 3221 0 0 0 2629
11 5888316 1514 0 0 4 4 753 753 0 0 0 52905
12 5831030 1426 0 0 0 0 713 713 0 0 0 8360
13 4114996 986 0 0 0 0 493 493 0 0 0 9929
14 5361222 1380 0 0 5 5 685 685 0 0 0 5844
15 38599148 9866 0 0 75 75 4858 4858 0 0 0 10241
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 33.879s ( 944.541 blocks/s, 59.034 Mb/s)
|
|
|
611
|
Fri Apr 26 23:04:51 2024 |
Marc | 00:00-08:00 Saturday 27 April |
It's a new day, it's a new shift and I'm feeeeeliiiiing goooood !!!!!
0.15 Checks:
DSSSD bias & leakage current - Grafana - attachments 1-2
leakage current dropped a bit since the evening shift ~22:00.
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 4
per FEE64 Rate spectra - attachment 5
Merger etc - attachment 6
ucesb - attachment 7
2:20 Nothing new to report. All good. Leakage currents continue to go down slowly. See attachement 8 to 12
3:30 Still running smoothly.
5:20 AIDA 4 stop counting in the last 20 min. I informed the main DAQ and they started a new run without AIDA.
DAQ restarted OK. Main DAQ informed.
|
| Attachment 1: Screenshot_from_2024-04-27_00-17-00.png
|
|
| Attachment 2: Screenshot_from_2024-04-27_00-16-16.png
|
|
| Attachment 3: Screenshot_from_2024-04-27_00-18-49.png
|
|
| Attachment 4: Screenshot_from_2024-04-27_00-19-05.png
|
|
| Attachment 5: Screenshot_from_2024-04-27_00-19-19.png
|
|
| Attachment 6: Screenshot_from_2024-04-27_00-21-44.png
|
|
| Attachment 7: Screenshot_from_2024-04-27_00-21-00.png
|
|
| Attachment 8: Screenshot_from_2024-04-27_02-32-22.png
|
|
| Attachment 9: Screenshot_from_2024-04-27_02-37-20.png
|
|
| Attachment 10: Screenshot_from_2024-04-27_02-37-31.png
|
|
| Attachment 11: Screenshot_from_2024-04-27_02-40-39.png
|
|
| Attachment 12: Screenshot_from_2024-04-27_02-41-13.png
|
|
|
|
610
|
Fri Apr 26 15:34:00 2024 |
RDP | 16:00 - 00:00 Friday 26 April |
This appeared in the system log:
Apr 26 16:30:51 aidas-gsi smartd[1076]: Device: /dev/sda [SAT], 1 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 2224 Currently unreadable (pending) sectors
Apr 26 16:30:56 aidas-gsi smartd[1076]: Device: /dev/sdd [SAT], 257 Offline uncorrectable sectors
These messages seem to appear sporadically.
18:10 Link 3 is giving rates of zero (see attachment 3). Is this normal?
18:15 daq error on ucesb (attachment 4). Zero rate through event builder. Emailed Nic & can see he's now on Zoom.
aida07 rebooted itself, but WR out of sequence still
manually reboot aida07, still out of sequence
do full powercycle of aida... now all FEEs are in sequence again. Resume running to disk and alert DESPEC shifters
Thanks Nic!
22:14 AIDA DSSD 2 HV status = 99 on grafana (attachment 6)
22:43 Noticed that aida02 ASIC temperature reading is now 0.00 instead of 511(!) that it was previously - see attachment 2 in #608. It was similarly high earlier in this shift too.
|
| Attachment 1: Screenshot_2024-04-26_at_16-40-03_ucesb.png
|
|
| Attachment 2: Screenshot_2024-04-26_at_16-41-57_AIDA_-_Grafana.png
|
|
| Attachment 3: Screenshot_2024-04-26_at_19-07-21_Merger_Input_Data_Link_Rates_aidas-gsi.png
|
|
| Attachment 4: Screenshot_2024-04-26_at_19-18-28_ucesb.png
|
|
| Attachment 5: Screenshot_2024-04-26_at_22-12-06_ucesb.png
|
|
| Attachment 6: Screenshot_2024-04-26_at_22-11-47_AIDA_-_Grafana.png
|
|
|
|
609
|
Fri Apr 26 08:15:59 2024 |
TD | [How To] Recover DAQ when one FEE64 stops sending data |
Occasionally, one (sometimes more) FEE64(s) will stop producing data whilst the DAQ is running *and* data is being transferred to the Merger and TapeServer.
Typical symptoms are:
- no data in Rate spectrum [workspace 2, display 1, browser tab 'Spectrum Browser']
- zero rate in 'ADC data item' statistics [workspace 2, display 1, browser tab 'Statistics']
- Merger data link inactive [workspace 3, display 1, browser tab 'NewMerger Control']
active data links are green, inactive links are red
N.B. data links are numbered from 0 to n-1, FEE64s are numbered from 1 to n
0) Please infrom the 'on shift' crew that you need to restart the AIDA DAQ
1) STOP DAQ [workspace 2, display 1, browser tab 'Run Control']
all FEE64s should change status from 'going' to 'stopped' *except* whichever FEE64(s) have a problem - typically their status is 'going' or 'error'
2) Restart Merger by selecting the Merger icon [workspace 3, display 2]
terminal window showing the startup will popup
3) Setup Merger by selecting 'SETUP' [workspace 3, display 1, browser tab 'NewMerger Control']
4) Start Merger by selecting 'GO' [workspace 3, display 1, browser tab 'NewMerger Control']
all data links will be inactive (red)
5) Reboot problem FEE64s
Launch a command terminal and telnet to the problem FEE64 for example
telnet aida04
Enter user 'root' and enter password. When logged in reboot the FEE64 using the command
reboot
6) Monitor boot using server system log [workspace 1, display 2, browser tab 'terminal']
you should see two messages showing the FEE64 mounting filesystems
wait c. three minutes for boot to complete
7) Restart DAQ
Select DAQ status 'Update' [workspace 2, display 1, browser tab 'Run Control']
Select DAQ status 'RESET' [workspace 2, display 1, browser tab 'Run Control']
Select DAQ status 'SETUP' [workspace 2, display 1, browser tab 'Run Control']
Disable histogramming by selecting 'Histogramming Disable all' [workspace 2, display 1, browser tab 'Run Control']
Disable waveforms by selecting 'Enable Waveforms Disable all' [workspace 2, display 1, browser tab 'Run Control']
Disable data transfer by selecting 'Data transfer Disable #1 all' [workspace 2, display 1, browser tab 'Run Control']
8) Restore ASIC settings
Select 'Save/Restore Settings' [workspace 2, display 1, browser tab 'Control']
Select the correct 'Data Base Key' [workspace 2, display 1, browser tab 'Module Settings']
Select 'Restore Settings' [workspace 2, display 1, browser tab 'Module Settings']
Wait 10s
9) Check ASIC settings
Select 'Check ASIC Control' [workspace 2, display 1, browser tab 'ASIC Control']
Wait for popup indicating completion of checks and dismiss
10) System wide checks
Select 'Synchronise the ASIC clocks' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Collect all WR status error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Collect all FPGA timestamp error counters ...' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check clock status' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check WR decoder status' [workspace 2, display 1, browser tab 'System Wide Checks']
Select 'Check FPGA timestamp errors' [workspace 2, display 1, browser tab 'System Wide Checks']
make a note or take a screenshot of any error messages observed
11) Start DAQ
Enable histogramming by selecting 'Histogramming Enable All' [workspace 2, display 1, browser tab 'Run Control']
Enable waveforms by selecting 'Enable Waveforms Enable All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
12) Check WR timestamps by selecting 'Collect all timestamps' [workspace 2, display 1, browser tab 'GSI White Rabbit Control']
readout times will be displayed for each FEE64
check that the readout times are monotonically increasing as you read down the list of readout times - if not, contact expert support
13) Stop DAQ
Start DAQ by selecting 'STOP' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ has *actually* stopped - if not repeat 'STOP
14) Enable data transfer to Merger and TapeServer and start DAQ
Enable data transfer by selecting 'Enable data transfer Enable #1 All' [workspace 2, display 1, browser tab 'Run Control']
Start DAQ by selecting 'GO' [workspace 2, display 1, browser tab 'Run Control']
check that DAQ is *actually* going - if not repeat 'GO'
15) Check data is being transferred to Merger and TapeServer
Select 'Reload' [workspace 3, display 1, browser tab 'NewMerger Control']
you should see a non-zero 'current merged data rate' and all links should show as green at least some of the time (requires observation over several reloads)
Select 'Reload' [workspace 3, display 1, browser tab 'TapeServer']
you should see 'Blocks written' and 'kb written' counters increase
16) Check AIDA data is being forwarded from the TapeServer to MBS [workspace 3, display 2, command terminal]
17) Restart should now have been successfully completed. Please inform the 'on shift' crew' that AIDA is forwarding data.
If not, please request expert assistance.
|
|
|
608
|
Fri Apr 26 07:01:26 2024 |
Norah | 08:00 - 16:00 Friday 26 April |
10:00 Checks:
All seems smooth.
DSSD bias and leakage current ( Attachment 1)
FEE64 temperatures (Attachment 2)
Per FEE64 Rate spectra (Attachment 3)
ADC data item stats -(Attachment 4)
Ucesb (Attachment 5)
10:21
Aida05 was zero not reading had to call Nic to fix it.
10:34
Now, Aida05 is running well.
.......................................................
12:00 Checks:
Everything appears to be going smoothly, and I took screenshot of them.
12:28, Aida02 showed as not reading , had to email Nic to fix it.
12:30, it resumed reading and returned to normal operation.
.........................................................
14:00 Checks:
Everything is going well , and I have captured screenshots of each one.
......................................................
16:00 Checks:
All seems smooth, and I took screenshot of them.
|
| Attachment 1: Screenshot_from_2024-04-26_10-00-39.png
|
|
| Attachment 2: Screenshot_from_2024-04-26_10-02-35.png
|
|
| Attachment 3: Screenshot_from_2024-04-26_10-04-40.png
|
|
| Attachment 4: Screenshot_from_2024-04-26_10-06-18.png
|
|
| Attachment 5: Screenshot_from_2024-04-26_10-09-52.png
|
|
|
|
607
|
Thu Apr 25 22:45:30 2024 |
Tom, Muneerah, Norah | Over night shift AIDA 26 Fri |
01:02
Aida04 was zero not reading had to call Tom to fix it.
02:00 checked all good and attached.
Its been running good after that till 7:30 again the same problem happened and Tom fixed it.
|
| Attachment 1: Screenshot_from_2024-04-25_23-42-15.png
|
|
| Attachment 2: Screenshot_from_2024-04-26_00-05-47.png
|
|
| Attachment 3: Screenshot_from_2024-04-26_00-05-15.png
|
|
| Attachment 4: Screenshot_from_2024-04-26_00-04-22.png
|
|
| Attachment 5: Screenshot_from_2024-04-26_02-00-17.png
|
|
| Attachment 6: Screenshot_from_2024-04-26_01-58-39.png
|
|
| Attachment 7: Screenshot_from_2024-04-26_01-58-39.png
|
|
| Attachment 8: Screenshot_from_2024-04-26_01-57-38.png
|
|
| Attachment 9: Screenshot_from_2024-04-26_01-57-01.png
|
|
|
|
606
|
Thu Apr 25 15:02:50 2024 |
Marc | 16:00 - 0:00 Thursday 25 April |
16:20:
Spill length:4s (was 3.5s before) - Extraction time 1.5s (see attachement #1)
16.45 Checks:
DSSSD bias & leakage current - Grafana - attachments 2
leakage current ramping, spill micro structure observable
FEE64 temperatures OK - attachment 3
ADC data item stats - attachment 5
per FEE64 Rate spectra - attachment 4
Merger etc - attachment 6
ucesb - attachment 7
XX.XX Checks
DSSSD bias & leakage current - Grafana - attachments X
FEE64 temperatures OK - attachment X
ADC data item stats - attachment X
per FEE64 Rate spectra - attachment X
Merger etc - attachment X
ucesb - attachment X |
| Attachment 1: Screenshot_from_2024-04-25_16-19-32.png
|
|
| Attachment 2: Screenshot_from_2024-04-25_16-45-19.png
|
|
| Attachment 3: Screenshot_from_2024-04-25_16-46-12.png
|
|
| Attachment 4: Screenshot_from_2024-04-25_16-46-41.png
|
|
| Attachment 5: Screenshot_from_2024-04-25_16-47-00.png
|
|
| Attachment 6: Screenshot_from_2024-04-25_16-52-18.png
|
|
| Attachment 7: Screenshot_from_2024-04-25_16-52-50.png
|
|
| Attachment 8: Screenshot_from_2024-04-25_17-23-36.png
|
|
|
|
605
|
Thu Apr 25 08:26:50 2024 |
TD | 08:00-16:00 Thursday 25 April |
09.56 DAQ continues data file S100_alpha/R18_387
FRS switching settings
DSSSD bias & leakage current OK - attachments 1
FEE64 temperatures OK - attachment 2
aida02 ASIC temp - known fault
All system wide checks OK *except* as shown - attachments 3-5
WR timestamps OK - attachment 6
ADC, PAUSE, RESUME & Correlation Scaler data time stats - attachments 7-10
per FEE64 Rate spectra - attachments 11
per p+n FEE64 1.8.L spectra - attachments 12-13
aida09 pulser peak width 57 ch FWHM
common x/y scale => peak height proxy for width => all p+n FEE64s similar pulser peak widths
per n+n FEE64 1.8.L spectra - attachment 14
per FEE64 1.8.H spectra - attachments 15-16
per FEE64 1.8.W spectra - 20us FSR - attachments 17-18
only aida08 & aida09 currently updating - ASIC check load and/or DAQ STOP/GO required?
Merger, TapeServer etc - attachments 19-21
BNC PB-5 settings - attachment 22
10.07 data file S100_alpha/R20
new run for new FRS setting
10.08 all histograms, stats, merger stats and data link stats zero'd
10.10 ADC data item stats (beam OFF) - attachment 23
11/16 < 20k, max 129k (aida08)
10.31 analysis of data file R18_349 (closed 06.00 this morning)
max deadtime 12% (aida04)
no timewarps
HEC data item rate 1.7kHz
11.00 tuning 168dy FRS setting
ucesb - attachment 24
spill length 3.5s, extraction time 1.5s
11.30 degraders now set to implant in AIDA DSSSDs
high rates expected
per FEE64 Rate spectra - attachment 25
ADC data item stats - attachment 26
Merger etc - attachment 27
in spill rates to MBS up 30+Mb/s, 2.5M data item/s merged
DSSSD bias & leakage current - attachment 28
c. few hundred nA step in leakage current
analysis of data file R20_33 - attachment 29
max deadtime 18% (aida08)
no timewarps
HEC data item rate 4.1kHz (c. 2.4x previous FRS setting)
Correcting for spill cycle => 5.9kHz in spill (online analysis shows no HEC events observed outside spill)
assume typical HEC event multiplicity = 3 => implant rate ~ 2.0kHz in spill or 6800 implants per spill
SC42 L/R currently report c. 4.5kHz
Note that we currently see a higher fraction of events in DSSSD#2 cf. previous setting so multiplicity assumption may be low
If we assume multipicity = 6 => implant rate ~ 1.0kHz in spill or 3200 implants per spill
we are probably somewhere in between
12.50 DSSSD bias & leakage current - Grafana - attachment 30
leakage current ramping, spill micro structure observable
FEE64 temperatures OK - attachment 31
ADC data item stats - attachment 32
per FEE64 Rate spectra - attachment 33
Merger etc - attachment 34
14.56 analysis of data file R20_106 - attachment 35
max deadtime c. 18% (aida04 & aida08), c. 12% (aida03), c. 11% (aida06), c. 9% (aida02 & aida13), c. 6% (aida05), all others <2%
no timewarps
HEC data item rate 4.2kHz
15.40 DSSSD bias & leakage current - Grafana - attachments 36-37
leakage current ramping, spill micro structure observable
FEE64 temperatures OK - attachment 38
ADC data item stats - attachment 39
per FEE64 Rate spectra - attachment 40
Merger etc - attachment 41
ucesb - attachment 42
15.57 all histograms zero'd
|
| Attachment 1: Screenshot_from_2024-04-25_09-27-20.png
|
|
| Attachment 2: Screenshot_from_2024-04-25_09-27-44.png
|
|
| Attachment 3: Screenshot_from_2024-04-25_09-28-05.png
|
|
| Attachment 4: Screenshot_from_2024-04-25_09-28-16.png
|
|
| Attachment 5: Screenshot_from_2024-04-25_09-28-23.png
|
|
| Attachment 6: Screenshot_from_2024-04-25_09-29-04.png
|
|
| Attachment 7: Screenshot_from_2024-04-25_09-29-20.png
|
|
| Attachment 8: Screenshot_from_2024-04-25_09-29-46.png
|
|
| Attachment 9: Screenshot_from_2024-04-25_09-29-59.png
|
|
| Attachment 10: Screenshot_from_2024-04-25_09-30-15.png
|
|
| Attachment 11: Screenshot_from_2024-04-25_09-30-56.png
|
|
| Attachment 12: Screenshot_from_2024-04-25_09-31-55.png
|
|
| Attachment 13: Screenshot_from_2024-04-25_09-32-11.png
|
|
| Attachment 14: Screenshot_from_2024-04-25_09-33-15.png
|
|
| Attachment 15: Screenshot_from_2024-04-25_09-34-23.png
|
|
| Attachment 16: Screenshot_from_2024-04-25_09-34-52.png
|
|
| Attachment 17: Screenshot_from_2024-04-25_09-35-38.png
|
|
| Attachment 18: Screenshot_from_2024-04-25_09-36-14.png
|
|
| Attachment 19: Screenshot_from_2024-04-25_09-36-28.png
|
|
| Attachment 20: Screenshot_from_2024-04-25_09-37-02.png
|
|
| Attachment 21: Screenshot_from_2024-04-25_09-37-17.png
|
|
| Attachment 22: Screenshot_from_2024-04-25_10-02-31.png
|
|
| Attachment 23: R18_349
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 254055879 ( 798496.4 Hz)
Other data format: 7864121 ( 24716.9 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 850 ( 2.7 Hz)
RESUME: 851 ( 2.7 Hz)
SYNC100: 33185 ( 104.3 Hz)
WR48-63: 33185 ( 104.3 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 7796050 ( 24502.9 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 547807 ( 1721.8 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 318.168 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.011 0.000
1 15.547 0.000
2 0.026 0.000
3 39.441 0.000
4 0.410 0.000
5 0.954 0.000
6 0.000 0.000
7 0.601 0.000
8 0.003 0.000
9 0.000 0.000
10 0.050 0.000
11 0.014 0.000
12 0.000 0.000
13 0.000 0.000
14 0.017 0.000
15 0.116 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 6649451 11230 0 0 3 3 839 839 0 9546 0 49433
1 33550123 378474 0 0 245 245 4373 4373 0 369238 0 100748
2 7942952 3356106 0 0 5 5 1429 1429 0 3353238 0 35304
3 50066846 2782463 0 0 379 380 6547 6547 0 2768610 0 28219
4 11634489 554162 0 0 19 19 1542 1542 0 551040 0 47658
5 18478696 749338 0 0 77 77 2403 2403 0 744378 0 37331
6 6034652 1500 0 0 0 0 750 750 0 0 0 11345
7 41275133 10854 0 0 71 71 5356 5356 0 0 0 92779
8 3808950 934 0 0 2 2 465 465 0 0 0 19548
9 5816078 1448 0 0 0 0 724 724 0 0 0 8443
10 16595017 4342 0 0 16 16 2155 2155 0 0 0 3675
11 5732285 1360 0 0 2 2 678 678 0 0 0 65221
12 7170732 1770 0 0 0 0 885 885 0 0 0 12515
13 5117877 1350 0 0 0 0 675 675 0 0 0 13502
14 5096608 1234 0 0 2 2 615 615 0 0 0 7339
15 29085990 7556 0 0 29 29 3749 3749 0 0 0 14747
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 31.840s ( 1005.030 blocks/s, 62.814 Mb/s)
|
| Attachment 24: Screenshot_from_2024-04-25_11-27-43.png
|
|
| Attachment 25: Screenshot_from_2024-04-25_11-39-59.png
|
|
| Attachment 26: Screenshot_from_2024-04-25_11-40-11.png
|
|
| Attachment 27: Screenshot_from_2024-04-25_11-40-28.png
|
|
| Attachment 28: Screenshot_from_2024-04-25_11-43-22.png
|
|
| Attachment 29: R20_33
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 257467881 ( 1603138.0 Hz)
Other data format: 4452119 ( 27721.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1792 ( 11.2 Hz)
RESUME: 1788 ( 11.1 Hz)
SYNC100: 32598 ( 203.0 Hz)
WR48-63: 32598 ( 203.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 4383343 ( 27293.1 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 659812 ( 4108.4 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 160.602 s
FEE elapsed dead time(s) elapsed idle time(s)
0 4.032 0.000
1 14.271 0.000
2 17.643 0.000
3 26.583 0.000
4 10.476 0.000
5 15.929 0.000
6 1.815 0.000
7 29.387 0.000
8 0.158 0.000
9 0.036 0.000
10 0.592 0.000
11 0.531 0.000
12 14.310 0.000
13 0.165 0.000
14 0.189 0.000
15 0.995 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 14126134 8441 0 0 70 70 1791 1791 0 4719 0 78615
1 16495785 288035 0 0 163 162 2094 2094 0 283522 0 105734
2 19988207 1651337 0 0 161 161 2732 2732 0 1645551 0 49977
3 24594867 1476087 0 0 215 214 3183 3183 0 1469292 0 26591
4 15580694 468076 0 0 119 119 1941 1941 0 463956 0 27262
5 22750595 522722 0 0 300 299 2910 2910 0 516303 0 62407
6 11629769 3045 0 0 37 36 1486 1486 0 0 0 11991
7 37179788 9710 0 0 273 273 4582 4582 0 0 0 86342
8 7628214 1994 0 0 9 9 988 988 0 0 0 1179
9 5725517 1418 0 0 5 5 704 704 0 0 0 1622
10 13190745 3550 0 0 69 69 1706 1706 0 0 0 70
11 9285908 2326 0 0 29 29 1134 1134 0 0 0 74999
12 16713447 4700 0 0 252 252 2098 2098 0 0 0 21646
13 6115493 1498 0 0 25 25 724 724 0 0 0 35370
14 8494444 2072 0 0 11 11 1025 1025 0 0 0 92
15 27968274 7108 0 0 54 54 3500 3500 0 0 0 75915
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 34.629s ( 924.083 blocks/s, 57.755 Mb/s)
|
| Attachment 30: Screenshot_from_2024-04-25_12-50-09.png
|
|
| Attachment 31: Screenshot_from_2024-04-25_12-50-16.png
|
|
| Attachment 32: Screenshot_from_2024-04-25_12-50-28.png
|
|
| Attachment 33: Screenshot_from_2024-04-25_12-50-35.png
|
|
| Attachment 34: Screenshot_from_2024-04-25_12-51-02.png
|
|
| Attachment 35: R20_106
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 257522900 ( 1655989.1 Hz)
Other data format: 4397100 ( 28275.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1842 ( 11.8 Hz)
RESUME: 1844 ( 11.9 Hz)
SYNC100: 32579 ( 209.5 Hz)
WR48-63: 32579 ( 209.5 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 4328256 ( 27832.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 659185 ( 4238.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 155.510 s
FEE elapsed dead time(s) elapsed idle time(s)
0 3.085 0.000
1 13.568 0.000
2 18.124 0.000
3 27.697 0.000
4 9.792 0.000
5 16.474 0.000
6 2.671 0.000
7 28.543 0.000
8 0.065 0.000
9 0.015 0.000
10 0.643 0.000
11 0.597 0.000
12 14.389 0.000
13 0.083 0.000
14 0.439 0.000
15 1.481 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 13689199 8238 0 0 65 65 1771 1771 0 4566 0 77953
1 16947527 287988 0 0 165 166 2170 2170 0 283317 0 104868
2 20356284 1630731 0 0 174 175 2797 2797 0 1624788 0 51128
3 23896908 1435934 0 0 213 214 3077 3077 0 1429353 0 26122
4 15647459 464207 0 0 117 117 2008 2008 0 459957 0 27024
5 22696175 532714 0 0 306 305 2914 2914 0 526275 0 64287
6 12236208 3082 0 0 42 42 1499 1499 0 0 0 11994
7 36272075 9624 0 0 259 259 4553 4553 0 0 0 84854
8 7567267 1906 0 0 4 4 949 949 0 0 0 1286
9 5993294 1442 0 0 3 3 718 718 0 0 0 1730
10 13223300 3544 0 0 82 82 1690 1690 0 0 0 53
11 9630981 2436 0 0 36 36 1182 1182 0 0 0 75223
12 16792007 4646 0 0 270 270 2053 2053 0 0 0 21202
13 6155506 1514 0 0 19 19 738 738 0 0 0 35333
14 8544527 2006 0 0 19 19 984 984 0 0 0 68
15 27874183 7088 0 0 68 68 3476 3476 0 0 0 76060
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 30.426s ( 1051.740 blocks/s, 65.734 Mb/s)
|
| Attachment 36: Screenshot_from_2024-04-25_15-39-07.png
|
|
| Attachment 37: Screenshot_from_2024-04-25_15-39-14.png
|
|
| Attachment 38: Screenshot_from_2024-04-25_15-39-36.png
|
|
| Attachment 39: Screenshot_from_2024-04-25_15-39-53.png
|
|
| Attachment 40: Screenshot_from_2024-04-25_15-40-30.png
|
|
| Attachment 41: Screenshot_from_2024-04-25_15-40-48.png
|
|
| Attachment 42: Screenshot_from_2024-04-25_15-43-08.png
|
|
|
|
604
|
Wed Apr 24 21:39:51 2024 |
TD | 00:00-08:00 Thursday 25 April |
22.43 24.4.24
DSSSD bias & leakage current OK - attachment 1
FEE64 temperatures OK - attachment 2
ADC data item stats - attachment 3
per FEE64 rate spectra - attachments 4
Merger etc - attachment 5
analysis of data file R18_266 - attachment 6
max dead time 12% (aida04)
no timewarps
HEC data item rate 1.7kHz
03.37 DSSSD bias & leakage current OK - attachments 7-8
FEE64 temperatures OK - attachment 9
ADC data item stats - attachment 10
per FEE64 rate spectra - attachments 11
Merger etc - attachment 12 |
| Attachment 1: Screenshot_from_2024-04-24_22-43-23.png
|
|
| Attachment 2: Screenshot_from_2024-04-24_22-43-58.png
|
|
| Attachment 3: Screenshot_from_2024-04-24_22-44-50.png
|
|
| Attachment 4: Screenshot_from_2024-04-24_22-45-35.png
|
|
| Attachment 5: Screenshot_from_2024-04-24_22-46-21.png
|
|
| Attachment 6: R18_266
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 254416606 ( 820824.6 Hz)
Other data format: 7503396 ( 24208.2 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 846 ( 2.7 Hz)
RESUME: 846 ( 2.7 Hz)
SYNC100: 33155 ( 107.0 Hz)
WR48-63: 33155 ( 107.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 7435394 ( 23988.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 524404 ( 1691.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 309.952 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.014 0.000
1 16.009 0.000
2 0.003 0.000
3 37.388 0.000
4 0.176 0.000
5 0.544 0.000
6 0.000 0.000
7 0.784 0.000
8 0.015 0.000
9 0.000 0.000
10 0.083 0.000
11 0.004 0.000
12 0.000 0.000
13 0.000 0.000
14 0.017 0.000
15 0.227 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 6529814 10932 0 0 4 4 812 812 0 9300 0 46302
1 32238460 350302 0 0 231 231 4129 4129 0 341582 0 92776
2 8209450 3208628 0 0 2 2 1421 1421 0 3205782 0 32990
3 48353626 2676061 0 0 374 374 6369 6369 0 2662575 0 27015
4 10911439 520528 0 0 9 9 1454 1454 0 517602 0 45169
5 18550590 703657 0 0 64 64 2488 2488 0 698553 0 35506
6 6051492 1578 0 0 0 0 789 789 0 0 0 10690
7 41812278 11146 0 0 85 85 5488 5488 0 0 0 97961
8 3594476 900 0 0 4 4 446 446 0 0 0 18281
9 5838926 1490 0 0 0 0 745 745 0 0 0 8125
10 18621964 4722 0 0 27 27 2334 2334 0 0 0 3637
11 5447469 1386 0 0 1 1 692 692 0 0 0 60590
12 7038926 1642 0 0 0 0 821 821 0 0 0 12101
13 5011795 1240 0 0 0 0 620 620 0 0 0 12487
14 5028304 1286 0 0 2 2 641 641 0 0 0 7019
15 31177597 7898 0 0 43 43 3906 3906 0 0 0 13755
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 32.508s ( 984.379 blocks/s, 61.524 Mb/s)
|
| Attachment 7: Screenshot_from_2024-04-25_03-37-40.png
|
|
| Attachment 8: Screenshot_from_2024-04-25_03-38-57.png
|
|
| Attachment 9: Screenshot_from_2024-04-25_03-39-38.png
|
|
| Attachment 10: Screenshot_from_2024-04-25_03-40-20.png
|
|
| Attachment 11: Screenshot_from_2024-04-25_03-40-52.png
|
|
| Attachment 12: Screenshot_from_2024-04-25_03-41-31.png
|
|
|
|
603
|
Wed Apr 24 18:44:05 2024 |
PP | 19:30 checks |
All seems smooth.
Screenshots attached. |
| Attachment 1: Screenshot_from_2024-04-24_19-34-58.png
|
|
| Attachment 2: Screenshot_from_2024-04-24_19-37-12.png
|
|
| Attachment 3: Screenshot_from_2024-04-24_19-40-36.png
|
|
| Attachment 4: Screenshot_from_2024-04-24_19-41-39.png
|
|
| Attachment 5: Screenshot_from_2024-04-24_19-42-20.png
|
|
| Attachment 6: Screenshot_from_2024-04-24_19-42-53.png
|
|
| Attachment 7: Screenshot_from_2024-04-24_19-43-21.png
|
|
| Attachment 8: Screenshot_from_2024-04-24_19-43-41.png
|
|
|
|
602
|
Wed Apr 24 15:37:59 2024 |
PP | 16:00-00:00 shift Wednesday 24 April |
All seems OK.
Screenshots attached. |
| Attachment 1: Screenshot_from_2024-04-24_16-38-45.png
|
|
| Attachment 2: Screenshot_from_2024-04-24_16-39-09.png
|
|
| Attachment 3: Screenshot_from_2024-04-24_16-39-38.png
|
|
| Attachment 4: Screenshot_from_2024-04-24_16-40-24.png
|
|
| Attachment 5: Screenshot_from_2024-04-24_16-41-15.png
|
|
| Attachment 6: Screenshot_from_2024-04-24_16-43-41.png
|
|
| Attachment 7: Screenshot_from_2024-04-24_16-44-15.png
|
|
| Attachment 8: Screenshot_from_2024-04-24_16-44-32.png
|
|