| |
ID |
Date |
Author |
Subject |
|
|
189
|
Thu Mar 11 07:09:25 2021 |
OH, LS | Thursday 11th of March |
08:00 System wide checks all ok *exctept aida07 and 10 on ADC calibration
N.B. wrong date given to the screenshots by accident (It's early)
Statistics - attachment 1
FEE temperatures (AIDA10 unavailable) - attachment 2
FEE temperature - attachment 3
Bias and leakage currents ok - attachment 4
08:34 The FRS stopped receiving ions. AIDA crashed simultaneously with it.
Have now recovered AIDA daq - Looking at the messages log AIDA04 restarted itself.
Restarted but no stats in aida04
It did not recover from a reset
Telnet in and did a reboot command
09:10 Finally recovered issue was it was not linking to merger
There is an options issue with aida12. The ASIC settings had become undefined and there are no histograms
Manually set the ASIC settings for aida12. Rates look as expected after
Will try to reset once beam is back
10:12 Problem with the DAQ so stopping AIDA while they work will switch MBS relay out to new version
Analysis of R30_21 looks fine
10:38 System wide checks ok
Statistics - attachment 5
Temperature - attachment 6
Bias and leakage - attachment 7
12.10 (LS)
System wide checks all okay, no fails
Statistics (attachment8)
Rate spectra (attachment9)
FEE temps (attament10)
Leakage currents written to sheet (attachment11)
Merger ~45M items/s
Tapeserver ~14MB/s
13.45 no beam (around file R30_98)
14.00 System wide checks all okay, no fails
Statistics (attachment12)
FEE temps (attament13)
Leakage currents written to sheet (attachment14)
Merger ~44M items/s
Tapeserver ~14MB/s
Beam still down
14.11 beam back aida file R30_112
14.13 beam back down, back at 14.16
14.20 no beam
14.23 notice some timestamp error in ucesb 37 minutes ago (about R30_102) no errors appeared on the MBS relay terminal,
double by analysing R30_101,102,103 no timewarps
beam is back but low intensity at the moment
14.46 beam looks to be back at a higher intensity now R30_127 (attachment15)
16.00 System wide checks all okay, no fails
Statistics (attachment16)
Rate spectra (attachment17)
FEE temps (attament18)
Leakage currents written to sheet (attachment19)
Merger ~45M items/s
Tapeserver ~14MB/s
18.00 System wide checks all okay, no fails
Statistics (attachment20)
Rate spectra (attachment21)
FEE temps (attament22)
Leakage currents written to sheet (attachment23)
Merger ~45M items/s
Tapeserver ~15MB/s
bunch of timewarp errors in ucesb, no errors in MBS relay terminal though
now see error in MBS relay!
"Warning: At least one MIDAS block missed in relay" Warning was set up by NH and OH to detect if the LSB of the first ADC data item in a block is less than the LAST LSB of the previous data block. It then iterates the info code 4 and if needed 5 stored before resetting the LSB stored.
It triggered 63 times.
I have checked all the files in the vicinity and there are no timewarps in the MIDAS data visible to TD's analyser
19:10 MBS DAQ stopped to allow FRS tests
19:38 MBS DAQ is back
20:15 Aida07 fails FPGA check
Base Current Difference
aida07 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
All other system wide checks ok
Statistics - attachment 24 (Again ignore the year in the filename)
Temperatures - attachment 25
Bias and leakage currents - 26
21:36 They are resetting the time sorter due to FATIMA issues
22:38 System wide checks error on WR in addition to the previous FPGA error
Base Current Difference
aida05 fault 0x5cc9 : 0x5cca : 1
aida06 fault 0xde76 : 0xde77 : 1
aida07 fault 0xa566 : 0xa567 : 1
aida08 fault 0x6ab0 : 0x6ab1 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Statistics - attachment 27
Temperature - attachment 28
Bias and leakage current - attachment 29
23:02 We still see timewarp errors in the ucesb. With our changes to the relay we can confirm that the WR headers in the MBS relay are time ordered. The issue must be downstream in the MBS chain. |
| Attachment 1: 210310_0805_Stats.png
|
|
| Attachment 2: 210310_0806_Temp.png
|
|
| Attachment 3: 210310_0806_Temp2.png
|
|
| Attachment 4: 210310_0808_Bias.png
|
|
| Attachment 5: 210311_1039_Stats.png
|
|
| Attachment 6: 210311_1039_Temp.png
|
|
| Attachment 7: 210311_1041_Bias.png
|
|
| Attachment 8: 1200_Statistics.PNG
|
|
| Attachment 9: 1200_SpectraRate.PNG
|
|
| Attachment 10: 1200_FEETemps.PNG
|
|
| Attachment 11: 1200_LeakageCurrents.PNG
|
|
| Attachment 12: 1400_Statistics.PNG
|
|
| Attachment 13: 1400_FEETemps.PNG
|
|
| Attachment 14: 1400_LeakageCurrents.PNG
|
|
| Attachment 15: 1450_SpectraRate.PNG
|
|
| Attachment 16: 1600_Statistics.PNG
|
|
| Attachment 17: 1600_SpectraRate.PNG
|
|
| Attachment 18: 1600_FEETemps.PNG
|
|
| Attachment 19: 1600_LeakageCurrents.PNG
|
|
| Attachment 20: 1800_Statistics.PNG
|
|
| Attachment 21: 1800_SpectraRate.PNG
|
|
| Attachment 22: 1800_FEETemps.PNG
|
|
| Attachment 23: 1800_LeakageCurrents.PNG
|
|
| Attachment 24: 200311_2014_Stats.png
|
|
| Attachment 25: 200311_2014_Temp.png
|
|
| Attachment 26: 210311_2017_Bias.png
|
|
| Attachment 27: 210311_2225_sTATS.png
|
|
| Attachment 28: 210311_2226_Temp.png
|
|
| Attachment 29: 210311_2227_Bias.png
|
|
|
|
144
|
Thu Mar 12 08:04:24 2020 |
OH, CA, TD | Thursday 12th March 08:00-16:00 |
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
09:00 Bias and leakage currents ok - attachment 1
Statistics ok - attachment 2
FEE temperatures ok - attachment 3
System wide checks all ok
09:08 No beam, not sure exactly how long it has been down.
FRS team are working on it
09:28 still no beam. issues with SIS.
DAQ set to NoStorage mode
most recent file R7_522
10.27 beam is back
writing to file R9_0
10.43 attachment 4 - hit rate spectra
11:09 all system wide checks ok
FEE64 temps ok - attachment 5
good event statistics ok - attachment 6
detector bias and leakage currents - attachment 7
11.18 merger ok, ~ 2.7 million data items per second
TapeServer ok, ~ 25 Mb/s data rate
Data forwarding to MBS ok
11.20 writing to file R9_39
11.22 beam off
11.24 beam back online (R9_42)
11.41 FRS has reverted to an older setting
13:04 hit rate spectra - attachment 8
13:19 all system wide checks ok
13:30 FEE64 temps ok - attachment 9
good event statistics ok - attachment 10
detector bias and leakage currents ok - attachment 11
15:31 good event statistics ok - attachment 13
detector bias and leakage currents ok - attachment 14
FEE64 temps ok - attachment 15
all system wide checks ok
beam seems to have stopped during 2pm meeting
15.35 TapeServer ok, data rate ~ 23 Mb/s
Merger ok, rate of 2.5 - 3 million data items per second
data forwarding to MBS ok
15.38 AIDA writing to file R9_233
16:02 beam back - AIDA writing to file R9_250
|
| Attachment 1: 200312_0902_bias.png
|
|
| Attachment 2: 200312_0902_stats.png
|
|
| Attachment 3: 200312_0903_temp.png
|
|
| Attachment 4: 200312_1042_rates.png
|
|
| Attachment 5: 200312_1105_temp.png
|
|
| Attachment 6: 200312_1107_stats.png
|
|
| Attachment 7: 200312_1108_stats.png
|
|
| Attachment 8: 200312_1302_rates.png
|
|
| Attachment 9: 200312_1321_temp.png
|
|
| Attachment 10: 200312_1325_stats.png
|
|
| Attachment 11: 200312_1327_bias.png
|
|
| Attachment 12: 200312_1528_stats.png
|
|
| Attachment 13: 200312_1529_bias.png
|
|
| Attachment 14: 300312_1527_temp.png
|
|
|
|
441
|
Fri May 13 06:56:14 2022 |
OH, BA, MA | Low Rates |
| Quote: |
|
The rate is very low in all aida, not sure .
|
This was because the statistics page had been changed to transfer buffers. The rates were ok |
|
|
430
|
Thu May 12 00:37:52 2022 |
OH, AM | Thursday 12 May |
01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
That is now complete.
Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc
R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
Will lower 2 and 4 to 200 and raise 6 and 8 to 300
R17 2 good, 4 shakey 6 and 8 poor - attachment 2
R18 2 and 4 at 220 and 6 and 7 at 350
2 and 8 low deadtime, 4 and 6 could be better. - attachment 3
Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
Statistics at this rate - attachment 4
02:18 Statistics - attachment 7
Temperatures - attachment 6
Bias & leakage current - attachment 5
Clock Status - OK
White Rabbit - OK
The other 3 checks currently aren't working
04:11 Statistics - attachment 10
Temperatures - attachment 9
Bias & leakage current - attachment 8
Clock Status - OK
White Rabbit - OK
06:05 Statistics - attachment 13
Temperatures - attachment 12
Bias & leakage current - attachment 11
Clock Status - OK
White Rabbit - OK |
| Attachment 1: R16_19_analyse.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 18197
ADC data format: 144240941 ( 1037416.0 Hz)
Other data format: 4701506 ( 33814.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 651 ( 4.7 Hz)
RESUME: 651 ( 4.7 Hz)
SYNC100: 18715 ( 134.6 Hz)
WR48-63: 18715 ( 134.6 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 4662774 ( 33535.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 9906 ( 71.2 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 139.039 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 96.976
1 1.224 0.000
2 0.000 2.784
3 4.445 0.000
4 0.000 108.587
5 78.372 0.000
6 0.000 48.067
7 67.687 0.000
8 0.000 100.043
9 0.000 102.969
10 0.000 122.226
11 0.000 21.170
12 0.000 2.713
13 0.000 116.657
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 346857 9766 0 0 0 0 47 47 0 9672 0 2994
1 28465912 7268 0 0 10 10 3624 3624 0 0 0 3890
2 235746 2352664 0 0 0 0 299 299 0 2352066 0 528
3 33196911 2283279 0 0 20 20 4471 4471 0 2274297 0 644
4 346134 72 0 0 0 0 36 36 0 0 0 4
5 37750759 9970 0 0 309 309 4676 4676 0 0 0 836
6 855292 17907 0 0 0 0 114 114 0 17679 0 522
7 38520306 19374 0 0 312 312 4845 4845 0 9060 0 313
8 312768 68 0 0 0 0 34 34 0 0 0 23
9 342945 90 0 0 0 0 45 45 0 0 0 14
10 176833 46 0 0 0 0 23 23 0 0 0 7
11 1559416 390 0 0 0 0 195 195 0 0 0 62
12 1848830 538 0 0 0 0 269 269 0 0 0 19
13 282232 74 0 0 0 0 37 37 0 0 0 50
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 18.576s ( 979.614 blocks/s, 61.226 Mb/s)
|
| Attachment 2: R17_19_analyse.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 9844
ADC data format: 78473161 ( 1119546.3 Hz)
Other data format: 2099979 ( 29959.6 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 470 ( 6.7 Hz)
RESUME: 470 ( 6.7 Hz)
SYNC100: 10105 ( 144.2 Hz)
WR48-63: 10105 ( 144.2 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 2078829 ( 29657.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 0 ( 0.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 70.094 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 61.805
1 4.886 0.000
2 0.000 0.000
3 17.776 0.000
4 0.000 55.325
5 34.711 0.000
6 0.000 20.226
7 27.353 0.000
8 0.000 57.388
9 0.000 53.302
10 0.000 59.261
11 0.000 7.690
12 0.000 11.252
13 0.000 58.364
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 137192 4907 0 0 0 0 16 16 0 4875 0 0
1 18540829 4668 0 0 40 40 2294 2294 0 0 0 0
2 107308 1183296 0 0 0 0 177 177 0 1182942 0 0
3 18083643 896022 0 0 126 126 2379 2379 0 891012 0 0
4 171162 40 0 0 0 0 20 20 0 0 0 0
5 19343787 5154 0 0 155 155 2422 2422 0 0 0 0
6 395908 126 0 0 0 0 63 63 0 0 0 0
7 19401210 5206 0 0 149 149 2454 2454 0 0 0 0
8 159836 40 0 0 0 0 20 20 0 0 0 0
9 172050 38 0 0 0 0 19 19 0 0 0 0
10 94357 20 0 0 0 0 10 10 0 0 0 0
11 792883 200 0 0 0 0 100 100 0 0 0 0
12 933311 234 0 0 0 0 117 117 0 0 0 0
13 139685 28 0 0 0 0 14 14 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 10.115s ( 973.186 blocks/s, 60.824 Mb/s)
|
| Attachment 3: R18_19_analyse.txt
|
*** TDR format 3.3.0 analyser - TD - May 2021
*** ERROR: READ I/O error: 5002
blocks: 9700
ADC data format: 70825654 ( 1030660.0 Hz)
Other data format: 8568846 ( 124694.5 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 268 ( 3.9 Hz)
RESUME: 268 ( 3.9 Hz)
SYNC100: 9956 ( 144.9 Hz)
WR48-63: 9956 ( 144.9 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 8548398 ( 124396.9 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 2 ( 0.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 68.719 s
FEE elapsed dead time(s) elapsed idle time(s)
0 0.000 40.039
1 0.526 0.000
2 0.044 0.000
3 16.252 0.000
4 0.000 41.343
5 16.542 0.000
6 0.000 37.115
7 4.958 0.000
8 0.000 62.160
9 0.000 48.394
10 0.000 57.328
11 0.000 39.469
12 0.000 0.000
13 0.000 54.952
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 123341 4797 0 0 0 0 9 9 0 4779 0 0
1 16458296 4022 0 0 5 5 2006 2006 0 0 0 0
2 67098 4844172 0 0 2 2 649 649 0 4842870 0 2
3 14877427 3705657 0 0 108 108 2346 2346 0 3700749 0 0
4 201730 54 0 0 0 0 27 27 0 0 0 0
5 18757776 4968 0 0 118 118 2366 2366 0 0 0 0
6 275379 62 0 0 0 0 31 31 0 0 0 0
7 18591077 4758 0 0 35 35 2344 2344 0 0 0 0
8 87837 12 0 0 0 0 6 6 0 0 0 0
9 156274 28 0 0 0 0 14 14 0 0 0 0
10 61114 18 0 0 0 0 9 9 0 0 0 0
11 292464 64 0 0 0 0 32 32 0 0 0 0
12 736144 204 0 0 0 0 102 102 0 0 0 0
13 139697 30 0 0 0 0 15 15 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 10.125s ( 958.025 blocks/s, 59.877 Mb/s)
|
| Attachment 4: 220512_0155_stats.png
|
|
| Attachment 5: 220512_0217_biasleakage.png
|
|
| Attachment 6: 220512_0215_temps.png
|
|
| Attachment 7: 220512_0214_stats.png
|
|
| Attachment 8: 220512_0410_biasLeak.png
|
|
| Attachment 9: 220512_0410_temps.png
|
|
| Attachment 10: 220512_0408_stats.png
|
|
| Attachment 11: 220512_0604_biasLeakage.png
|
|
| Attachment 12: 220512_0603_temps.png
|
|
| Attachment 13: 220512_0601_stats.png
|
|
|
|
290
|
Tue May 4 12:11:39 2021 |
OH TD | Bias test of triple |
13:11 System wide checks all ok except *aida02 fails ADC calibration*
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Tue May 04 10:57:38 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Thu Apr 29 14:44:05 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
ASIC clocks have been synchronised and ADC re-calibrated
Rate spectra - attachment 1
Layout7 - attachment 2
Layout8 - attachment 3
Detectors at 130V
Statistics - attachment 4
V-I Curve - attachment 5
Voltage Ch0 Ch1
10 1.345 1.49
20 2.01 2.285
30 2.405 2.8
40 2.635 3.13
50 2.82 3.37
60 2.95 3.57
70 3.08 3.725
80 3.185 3.865
90 3.26 3.98
100 3.33 4.105
110 3.42 4.22
120 3.525 4.35
130 3.64 4.46
140 3.79 4.37
150 3.98 4.37
150+10min 3.9 4.485 |
| Attachment 1: 210504_1325_Rate.png
|
|
| Attachment 2: 210504_1326_Layout7.png
|
|
| Attachment 3: 210504_1327_Layout08.png
|
|
| Attachment 4: 210504_1334_Stats.png
|
|
| Attachment 5: VICurve.png
|
|
|
|
199
|
Sun Mar 14 08:16:00 2021 |
OH PJCS | Sunday 14th March 2021 |
> 08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151
>
> Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
> It is also the least noisy FEE in the back DSSD
>
> MERGE Data Link (20510): bad timestamp 10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
> MERGE Data Link (20510): bad timestamp 10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262
>
> 09:01 System wide checks
>
> Base Current Difference
> aida07 fault 0x8c74 : 0x8c77 : 3
> White Rabbit error counter test result: Passed 11, Failed 1
>
> Understand the status reports as follows:-
> Status bit 3 : White Rabbit decoder detected an error in the received data
> Status bit 2 : Firmware registered WR error, no reload of Timestamp
> Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
>
> Base Current Difference
> aida07 fault 0x2 : 0x5 : 3
> FPGA Timestamp error counter test result: Passed 11, Failed 1
> If any of these counts are reported as in error
> The ASIC readout system has detected a timeslip.
> That is the timestamp read from the time FIFO is not younger than the last
>
> Statistics - attachment 1
> Temperature - attachment 2
> Bias and leakage current - attachment 3
It might be worth considering decreasing the time between the flushing of slower FEEs. Is the Merger waiting for the slower data and causing ‘deadtime’ in the faster ones. Perhaps consider this as a
balancing across the system? |
|
|
487
|
Tue Jun 21 10:19:05 2022 |
OH ML | Tuesday 21st June 08:00-00:00 |
11:19 Stats ok 0xa on all FEE - attachment 1
Current pulser settings - attachment 2
FEE Temperatures ok - attachment 3
Noticed both AIDA03 and AIDA07 had ASICs with missing channels. Did a check ASIC control on them to recover
Layout 1 after recovering channels - attachment 4
New stats (Note increased rate on 3 and 7) attachment 5
Bias and leakage currents ok - attachment 6
S4 now in closed access. Trying to find out if the beam dump has been put in front of AIDA. Will make sure this is checked before beam to S4.
14:05 Have had it confirmed that the flange beam dump is in position in S4
15:43 Stats ok -attachment 7
Temp ok - attachment 8
Bias and Leakage current ok - attachment 9
17:45 Currently the time machine trigger(for correlation) is connected to the DTAS or.
As the beam is tuning they are triggering at 150k. This is 450k worth of scalers in AIDA
During the experiment this will be changed to SC1 trigger to solve this.
19:49: Stats ok - attachment 10
temperatures ok - attachment 11
Bias - Leakage current ok - attachment 12 |
| Attachment 1: 220621_1119_Stats.png
|
|
| Attachment 2: 220621_1120_Pulser.png
|
|
| Attachment 3: 220621_1120_Temp.png
|
|
| Attachment 4: 220621_1123_Layout1.png
|
|
| Attachment 5: 220621_1123_Stats.png
|
|
| Attachment 6: 220621_1123_Bias.png
|
|
| Attachment 7: 220621_1542_Stats.png
|
|
| Attachment 8: 220621_1542_Temp.png
|
|
| Attachment 9: 220621_1543_Bias.png
|
|
| Attachment 10: Stats-Screenshot_from_2022-06-21_19-50-19.png
|
|
| Attachment 11: Temp-Screenshot_from_2022-06-21_19-48-56.png
|
|
| Attachment 12: HV_LC-Screenshot_from_2022-06-21_19-47-49.png
|
|
|
|
6
|
Thu Sep 20 08:57:48 2018 |
OH | Thursday 20th September |
09:57 Alpha run stoppped R2_0 -> R2_2
Thresholds dropped from 0x64 to 0xa
Pulser turned on again
System wide checks ok
FEE Temp ok - attachment 1
Bias ok, leakage at 1.485uA at 160V
Stats in similar place to yesterday - attachment 2
Low Energy pulser walkthrough
1.0V to 0.4V in 0.1V steps
2 minutes per step
Pulser at 50Hz
10:29 Run started R3_0
10:45 Run stopped R3_2
1.8.L spectra for low energy pulser walkthrough - attachment 3
11:00
High Energy pulser walkthrough
1.8V to 0.V in 0.2V
2 minutes per step
Pulser at 50Hz
11:06 Run started R4_0
10:18 Run stopped R4_2
1.8.L spectra for high energy pulser walkthrough - attachment 4 |
| Attachment 1: 180920_0958_Temp.png
|
|
| Attachment 2: 180920_1007_Stats.png
|
|
| Attachment 3: 180920_1042_LowEPulser.png
|
|
| Attachment 4: 180920_1117_HighEPulser.png
|
|
|
|
138
|
Tue Mar 10 10:16:56 2020 |
OH | AIDA Implant Range Scan |
A range scan was performed using AIDA.
See the GSI elog: https://elog.gsi.de/despec/S480/19 |
|
|
143
|
Wed Mar 11 23:44:27 2020 |
OH | Thursday 12th March 00:00-08:000 |
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
00:44 Implantation range in AIDA - attachment 1
01:14 At current rate will run out of space in 35 hours ~ 18:00 hours on Friday 13th March.
During the day if possible it might be wise to try and obtain a USB3 HDD mount to mount one of the spare drives in
the AIDA boxes and begin moving files onto that
01:35 System wide checks all ok
Bias and leakage current ok - attachment 2
Stats all ok - attachment 3
FEE temperatures ok - attachment 4
TapeServer rate around 25-29MB per second
Merger rate around 2.6-3.2 million items per second
04:11 System wide checks all ok
Bias and leakage currents ok - attachment 5
Stats all ok - attachment 6
FEE temp ok - attachment 7
06:38 System wide checks all ok
Bias and leakage currents ok - attachment 8
Stats all ok - attachment 9
FEE temp ok - attachment 10 |
| Attachment 1: 200311_ImpDis.png
|
|
| Attachment 2: 200312_0135_Bias.png
|
|
| Attachment 3: 200312_0136_Stats.png
|
|
| Attachment 4: 200312_0139_FEETemp.png
|
|
| Attachment 5: 200312_0405_BIAS.png
|
|
| Attachment 6: 200312_0405_stats.png
|
|
| Attachment 7: 200312_0406_Temp.png
|
|
| Attachment 8: 200312_0638_bias.png
|
|
| Attachment 9: 200312_0639_stats.png
|
|
| Attachment 10: 200312_0639_temp.png
|
|
|
|
169
|
Thu Mar 4 10:13:48 2021 |
OH | AnyDesk connection instuction |
|
| Attachment 1: Connecting_to_AnyDesk_at_GSI_for_AIDA.pdf
|
|
|
|
171
|
Thu Mar 4 11:16:13 2021 |
OH | Further cases of possible database issues |
During testing for the S452 experiment yesterday we were running through the steps of a restart.
The FEEs were not powercycled at this stage but the MIDAS server was restarted.
Initial setup went smoothly and no issues were encountered during setup.
NewMerger and TapeServer were both setup and set to going.
Upon ''Going'' the run control the data transfer for aida09 dropped out.
Going to the NewMerger it could be seen that data was making it through and data items were being merged.
The tapeserver however was seeing no data and neither was the MBS dataspy.
During this time the FEEs became unresponsive likely as their buffers filled. Trying to reset gave the attached errors.
Checking the folder manually it could be seen that the Options file was there.
At this point a powercycle was pewrformed but upong 'Going' the DAQ the same issue was again encountered.
This time we were able to recover control access to the FEEs by running multiple NewMerger sessions which helped clear the buffers.
During our search for the problem we looked at the Options from within MIDAS and could not see anything out of the ordinary.
We also tried restoring the Options from within MIDAS.
We again reached the same issue with the tapeserver.
At this point we restored the Options folder from by copying manually within terminal.
Following this we were able to start the DAQ normally. |
| Attachment 1: ErrorMessage.PNG
|
|
| Attachment 2: errorMessage.txt
|
EXPERIMENTS/AIDA/Options/aida04 TS_SYNC_PHASE NOT available: error 0x10004; value TS_SYNC_PHASE does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ExtClk NOT available: error 0x10004; value ExtClk does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.shift NOT available: error 0x10004; value Aida.shift does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 MACB_TRIG_MODE NOT available: error 0x10004; value MACB_TRIG_MODE does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.offset NOT available: error 0x10004; value Aida.offset does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 RunNumber NOT available: error 0x10004; value RunNumber does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_GroupBase NOT available: error 0x10004; value Aida_GroupBase does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Rate.channels NOT available: error 0x10004; value Rate.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Stat.channels NOT available: error 0x10004; value Stat.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.Vchannels NOT available: error 0x10004; value Aida.Vchannels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ASIC.settings NOT available: error 0x10004; value ASIC.settings does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_D_Enable NOT available: error 0x10004; value Aida_Hist_D_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 WAVE_DMA_HWM NOT available: error 0x10004; value WAVE_DMA_HWM does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Stat.shift NOT available: error 0x10004; value Stat.shift does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.Wchannels NOT available: error 0x10004; value Aida.Wchannels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_V_Enable NOT available: error 0x10004; value Aida_Hist_V_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Include.Aida NOT available: error 0x10004; value Include.Aida does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida.channels NOT available: error 0x10004; value Aida.channels does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 DataAcqPgm NOT available: error 0x10004; value DataAcqPgm does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_H_Enable NOT available: error 0x10004; value Aida_Hist_H_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 DataFormat NOT available: error 0x10004; value DataFormat does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 ASIC_DMA_HWM NOT available: error 0x10004; value ASIC_DMA_HWM does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
EXPERIMENTS/AIDA/Options/aida04 Aida_Hist_L_Enable NOT available: error 0x10004; value Aida_Hist_L_Enable does not exist in node /MIDAS/DB/EXPERIMENTS/AIDA/Options/aida04
STATE for aida01 returned with an error
error: SOAP http transport timed out after 20000 ms
NONE
error: SOAP http transport timed out after 20000 ms
while executing
"$transport $procVarName $url $req"
(procedure "::SOAP::invoke" line 18)
invoked from within
"::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState"
("eval" body line 1)
invoked from within
"eval ::SOAP::invoke ::SOAP::_DataAcquisitionControlClient__GetState $args"
(procedure "DataAcquisitionControlClient__GetState" line 1)
invoked from within
"DataAcquisitionControlClient__GetState"
|
|
|
172
|
Fri Mar 5 22:53:05 2021 |
OH | Thursday 5th of March |
There has been no beam for much of the day. It was lost at 13:00 and not regained until 22:00 German time.
At some point during the day the White Rabbit timestamp cable was unplugged from AIDA. This caused it lose correlation with the rest of the system.
At first a reset and setup was performed to try and regain correlation but that did not prove successful.
A full powercycle was then performed at ~22:30 German time which upon restarting the DAQ solved the issue.
The plan is to overnight run some fragments, for the purposes of observing the isomers. This data is not essential and as such we will not be writing to disk.
Beam plug is in front of AIDA. This is causing a huge spike in the Ge. |
|
|
178
|
Sun Mar 7 23:13:50 2021 |
OH | Monday 8th March |
00:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:14 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
01:20 Can see now that we are writing to a separate drive the compression is going much faster - attachment 4
02:15 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 5
FEE Temperatures - attachment 6
Bias and leakage currents ok - attachment 7
03:43 Beam has been lost. Not and FRS fault. Several powersupplies have tripped
04:00 Noticed more bad merge events
System wide checks has also noted an FPGA error.
All other system wide checks same results as previous
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A3F29 BA0DF4D2 , WR/10=> 23DD31DC5CE3215, Readout Time => 23DD31DC6F34000
aida02 : White Rabbit=> 166A3F29 CC49ECD7 , WR/10=> 23DD31DC7A0FE15, Readout Time => 23DD31DC8428000
aida03 : White Rabbit=> 166A3F29 DF55B7E8 , WR/10=> 23DD31DC9889264, Readout Time => 23DD31DCA85C000
aida04 : White Rabbit=> 166A3F29 EECF82B7 , WR/10=> 23DD31DCB14C045, Readout Time => 23DD31DCC2B8000
aida05 : White Rabbit=> 166A3F29 FED2EFD2 , WR/10=> 23DD31DCCAEB195, Readout Time => 23DD31DCDC14000
aida06 : White Rabbit=> 166A3F2A 0EDC023D , WR/10=> 23DD31DCE49336C, Readout Time => 23DD31DCF204000
aida07 : White Rabbit=> 166A3F2A 1E8170D3 , WR/10=> 23DD31DCFD9BE7B, Readout Time => 23DD31DD10A0000
aida08 : White Rabbit=> 166A3F2A 30448F76 , WR/10=> 23DD31DD1A074BF, Readout Time => 23DD31DD2A34000
aida09 : White Rabbit=> 166A3F2A 3FE36E61 , WR/10=> 23DD31DD33057D6, Readout Time => 23DD31DD3B94000
aida10 : White Rabbit=> 166A3F2A 4E83A1F7 , WR/10=> 23DD31DD4A6C365, Readout Time => 23DD31DD46A8000
aida11 : White Rabbit=> 166A3F2A 5FD19086 , WR/10=> 23DD31DD661C1A7, Readout Time => 23DD31DD7664000
aida12 : White Rabbit=> 166A3F2A 6FE06F3D , WR/10=> 23DD31DD7FCD7EC, Readout Time => 23DD31DD9684000
04:07 Beam is back AIDA on file R6_815
Statistics - attachment 8
FEE Temperatures - attachment 9
Bias and leakage currents ok - attachment 10
ASIC check ok
Multiple merge data error events observed
06:04 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 11
FEE Temperatures - attachment 12
Bias and leakage currents ok - attachment 13
07:22 Beam noticed dropped but back quickly |
| Attachment 1: 210308_0014_Stats.png
|
|
| Attachment 2: 2303108_0015_Temp.png
|
|
| Attachment 3: 210308_0016_Bias.png
|
|
| Attachment 4: 210308_0120_System.png
|
|
| Attachment 5: 210308_0213_Stats.png
|
|
| Attachment 6: 210308_0213_Temp.png
|
|
| Attachment 7: 210308_0215_Bias.png
|
|
| Attachment 8: 210308_0403_Stats.png
|
|
| Attachment 9: 210308_0404_Temp.png
|
|
| Attachment 10: 210308_04011_Bias.png
|
|
| Attachment 11: 210308_0602_Stats.png
|
|
| Attachment 12: 210308_0603_Temp.png
|
|
| Attachment 13: 210308_0604_Bias.png
|
|
|
|
182
|
Mon Mar 8 22:51:47 2021 |
OH | Tuesday 9th March |
18:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
00:12 Noticed this log in UCESB.
Closed connection [140.181.60.97]...
1 clients...
AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848)
AIDA timewarp is over, skipped 2728 AIDA event(s)
=> Not emitting timewarped event (before 166a7a92fdac5c02)
Recovered from timewarp but skipped 5 event(s)
at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961
Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either
Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message
01:09 Another timewarp message
AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974)
=> Not emitting timewarped event (before 166a83bc6226e9ea)
Recovered from timewarp but skipped 6 event(s)
AIDA timewarp is over, skipped 323841 AIDA event
Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175
Again no timewarps in analysis of file.
02:00 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 4
FEE Temperatures - attachment 5
Bias and leakage currents ok - attachment 6
03:10 Problem with accelerator will be no beam for ~30 minutes
03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place.
There are now also errors in all FEEs for WR events
Base Current Difference
aida01 fault 0xc45f : 0xc46a : 11
aida02 fault 0xd250 : 0xd25b : 11
aida03 fault 0x27b2 : 0x27bd : 11
aida04 fault 0x714b : 0x7156 : 11
aida05 fault 0x74f5 : 0x7500 : 11
aida06 fault 0x558a : 0x5595 : 11
aida07 fault 0x8d20 : 0x8d2b : 11
aida08 fault 0x8131 : 0x813c : 11
aida09 fault 0xf412 : 0xf41d : 11
aida10 fault 0x82c2 : 0x82cd : 11
aida11 fault 0x2845 : 0x2850 : 11
aida12 fault 0x1bf6 : 0x1c01 : 11
White Rabbit error counter test result: Passed 0, Failed 12
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Also errors in FPGA
Base Current Difference
aida03 fault 0x0 : 0x1 : 1
aida04 fault 0x0 : 0x1 : 1
aida10 fault 0x0 : 0x2 : 2
aida11 fault 0x0 : 0x1 : 1
aida12 fault 0x1 : 0x2 : 1
FPGA Timestamp error counter test result: Passed 7, Failed 5
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A8D22 3A5FDE66 , WR/10=> 23DDAE9D2A32FD7, Readout Time => 23DDAE9D3C9C000
aida02 : White Rabbit=> 166A8D22 4C6C3AEB , WR/10=> 23DDAE9D4713917, Readout Time => 23DDAE9D5B94000
aida03 : White Rabbit=> 166A8D22 5FB33A3C , WR/10=> 23DDAE9D65EB906, Readout Time => 23DDAE9D769C000
aida04 : White Rabbit=> 166A8D22 6FBA29DB , WR/10=> 23DDAE9D7F9042F, Readout Time => 23DDAE9D8D14000
aida05 : White Rabbit=> 166A8D22 7F71248D , WR/10=> 23DDAE9D98B5074, Readout Time => 23DDAE9DA9D8000
aida06 : White Rabbit=> 166A8D22 8F78B071 , WR/10=> 23DDAE9DB25AB3E, Readout Time => 23DDAE9DC280000
aida07 : White Rabbit=> 166A8D22 9F5A64B7 , WR/10=> 23DDAE9DCBC3D45, Readout Time => 23DDAE9DDFF0000
aida08 : White Rabbit=> 166A8D22 B1E91092 , WR/10=> 23DDAE9DE974E75, Readout Time => 23DDAE9DFA08000
aida09 : White Rabbit=> 166A8D22 C150D9ED , WR/10=> 23DDAE9E021AF64, Readout Time => 23DDAE9E0FD8000
aida10 : White Rabbit=> 166A8D22 D02BA583 , WR/10=> 23DDAE9E19DF6F3, Readout Time => 23DDAE9E2D50000
aida11 : White Rabbit=> 166A8D22 E287CFC2 , WR/10=> 23DDAE9E373FB2D, Readout Time => 23DDAE9E4720000
aida12 : White Rabbit=> 166A8D22 F2D4C911 , WR/10=> 23DDAE9E515474E, Readout Time => 23DDAE9E1BCC000
System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results
AIDA DAQ Stopped (Still no beam)
The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA).
03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully.
All system wide checks in AIDA now pass. (AIDA06 no longer shows the error)
Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again.
05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7
The problem was with the accelerator operators system. There was no beam.
06:20 Error in WR system wide checks. Rest all ok
This time there is no bad merge items recorded in the merger terminal.
There are also no timewarp events seen in ucesb which were observed with the issues at 3am.
Base Current Difference
aida01 fault 0xdbe9 : 0xdbea : 1
aida02 fault 0xa93d : 0xa93e : 1
aida03 fault 0x8fc4 : 0x8fc5 : 1
aida04 fault 0x3e56 : 0x3e57 : 1
aida05 fault 0xb04c : 0xb04d : 1
aida06 fault 0x5576 : 0x5577 : 1
aida07 fault 0x99ff : 0x9a00 : 1
aida08 fault 0x8412 : 0x8413 : 1
aida09 fault 0x2457 : 0x2458 : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A9523 11478D1D , WR/10=> 23DDBB6B4ED8E1C, Readout Time => 23DDBB6B6148000
aida02 : White Rabbit=> 166A9523 236AA5E8 , WR/10=> 23DDBB6B6BDDD64, Readout Time => 23DDBB6B8164000
aida03 : White Rabbit=> 166A9523 375F235D , WR/10=> 23DDBB6B8BCB6BC, Readout Time => 23DDBB6B9C3C000
aida04 : White Rabbit=> 166A9523 47B8038C , WR/10=> 23DDBB6BA5F338E, Readout Time => 23DDBB6BA77C000
aida05 : White Rabbit=> 166A9523 599FC116 , WR/10=> 23DDBB6BC29934F, Readout Time => 23DDBB6BD34C000
aida06 : White Rabbit=> 166A9523 696EB0A7 , WR/10=> 23DDBB6BDBE44DD, Readout Time => 23DDBB6BECB8000
aida07 : White Rabbit=> 166A9523 7982236B , WR/10=> 23DDBB6BF59D057, Readout Time => 23DDBB6C0418000
aida08 : White Rabbit=> 166A9523 87C85008 , WR/10=> 23DDBB6C0C73B34, Readout Time => 23DDBB6C1C7C000
aida09 : White Rabbit=> 166A9523 97020497 , WR/10=> 23DDBB6C24D0075, Readout Time => 23DDBB6C34B4000
aida10 : White Rabbit=> 166A9523 A69B171D , WR/10=> 23DDBB6C3DC4F1C, Readout Time => 23DDBB6C50A0000
aida11 : White Rabbit=> 166A9523 B86C8D7D , WR/10=> 23DDBB6C5A4748C, Readout Time => 23DDBB6C6C88000
aida12 : White Rabbit=> 166A9523 CA1DD1B2 , WR/10=> 23DDBB6C76961C5, Readout Time => 23DDBB6C5C40000
Possibly occured at 05:28 as a ucesb reports a timewarp at the time.
AIDA Timewarp (166a922b75368928 before 166a922b799e6c92)
AIDA timewarp is over, skipped 2728 AIDA event(s)
Cannot check file as was not writing to file.
Statistics - attachment 8
FEE Temperatures - attachment 9
Bias and leakage currents ok - attachment 10
06:40 There are occasional seconds of beam. According to operators it comes and then it goes again
06:45 Beam fairly stable so we are running again.
AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19
07:00 Beam stopped again
07:02 Beam returned
There are apparently issues with MUSIC chamber one and its resolution
07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing. |
| Attachment 1: 210308_2354_Stats.png
|
|
| Attachment 2: 210308_2355_Temp.png
|
|
| Attachment 3: 210308_2357_Bias.png
|
|
| Attachment 4: 210309_0104_Stats.png
|
|
| Attachment 5: 210309_0205_Temp.png
|
|
| Attachment 6: 210309_0205_Bias.png
|
|
| Attachment 7: 210309_0529_layout1.png
|
|
| Attachment 8: 210309_0619_Stats.png
|
|
| Attachment 9: 210309_0620_Temp.png
|
|
| Attachment 10: 210309_0623_Bias.png
|
|
|
|
194
|
Fri Mar 12 23:56:00 2021 |
OH | Saturday March 13th 00:00-08:00 |
24:00 Joined in with TD to debug the dropout of AIDA05 - See https://elog.ph.ed.ac.uk/DESPEC/193
After both powercycles and telnet reboots we were still seeing 0 in the statistics but did note that the counter was > 0
The last check we did was a restart of the merger server. This solved the issue.
It is likely that the link to the FEE was dropped but not re-established upon the resets.
There was no error message that this was seen. From now on I recommend refreshing the statistics ever 30 minutes.
01:29 System wide check
N.B. May not have reset baselines after reset
WR fault
Base Current Difference
aida01 fault 0x1ad : 0x1af : 2
aida02 fault 0x4434 : 0x4436 : 2
aida03 fault 0xcf95 : 0xcf99 : 4
aida04 fault 0x25aa : 0x25ae : 4
aida05 fault 0x4b4 : 0x4b7 : 3
aida06 fault 0x834 : 0x837 : 3
aida07 fault 0x146d : 0x1472 : 5
aida08 fault 0xf4a8 : 0xf4ab : 3
aida09 fault 0x4f6f : 0x4f73 : 4
aida10 fault 0x7504 : 0x7506 : 2
aida11 fault 0x26fa : 0x26fc : 2
aida12 fault 0x2858 : 0x285b : 3
White Rabbit error counter test result: Passed 0, Failed 12
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Check
Base Current Difference
aida12 fault 0x0 : 0x4 : 4
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
01:42 Statistics - attachment 1
Temperature - attachment 2
Bias and leakage current - attachment 3
03:57 System wide checks
Base Current Difference
aida07 fault 0x1472 : 0x1473 : 1
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida12 fault 0x4 : 0xc : 8
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Statistics - attachment 4
Temp - attachment 5
Bias - attachment 6
05:27 System wide checks
Base Current Difference
aida07 fault 0x1472 : 0x1474 : 2
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0x0 : 0x1 : 1
aida12 fault 0x4 : 0x16 : 18
FPGA Timestamp error counter test result: Passed 10, Failed 2
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Statistics - attachment 7
Temp - attachment 8
Bias - attachment 9
05:51 Looking into the bad timestamp messages in the merger e.g.
MERGE Data Link (28348): bad timestamp 5 3 0xc14f8415 0x00eea280 0x0000cd2e70eea280 0x166bcd2e70eea280 0x166bcd2e716c1f80
Looking in the merger source for the bad timestamp message:
if (op->Time < LastTimeStamp) {
// invalid time stamp
(*StatsMem[TSSEQERR])++;
(*StatsMem[TSSEQERR+((LinkNum+1)*MAXCOUNTERS)])++;
sprintf(message_buffer, "bad timestamp %d %d 0x%08lx 0x%08lx 0x%016llx 0x%016llx 0x%016llx",LinkNum, link_table[LinkNum]->link_state, op->Data, op->Timestamp, INFO4, op->Time, LastTimeStamp);
report_message(MSG_WARNING); /***************************/
// LastTimeStamp = op->Time;
So the message is generated when the merger detects a timewarp.
Took the first warning a data block of errors (The first instance of that particular `LastTimeStamp` and c alculated the time difference between the new timestamp and the last timestamp
The time and LastTimestamp were 0x166bccef3f424a50 0x166bccef4f4232e0 respectively
The time difference between them is 268429456 which is 268ms which seems quite a large difference
06:39 AIDA Crashed
This time all FEEs were responsive but not showing any stats
When stopping the DAQ all FEEs except aida06 stopped - attachment 10
Did a reset of the DAQ and all recovered but no stats on aida06 - attachment 11
Regained DAQ with a powercycle and a complete restart of the AIDA:8115 Merger and TaperServer
It is worth noting that aida06 is connected to link 5 the data link which had been producing the bad merge messages overnight.
We have now had it in aida05, aida6 and aida07. Could it be to do with the correlation scaler rate going into these FEES?
Going through the var/log/messages on aida-3 aida06 rebooted itself at 06:37
Mar 13 06:37:16 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:918 for /home/Embedded/XilinxLinux/ppc_4xx/rfs/aida06 (/home/Embedded/XilinxLinux/ppc_4xx/rfs)
Mar 13 06:37:18 aidas-gsi xinetd[4578]: START: time-stream pid=0 from=::ffff:192.168.11.6
Mar 13 06:37:32 aidas-gsi rpc.mountd[4497]: authenticated mount request from 192.168.11.6:862 for /home/npg/MIDAS_Releases/23Jan19/MIDAS_200119 (/home/npg/MIDAS_Releases/23Jan19/MIDAS_200119)
Looking in /var/log/messages on aida06 no evidence of a reason why:
Mar 12 23:30:56 aida06 kernel: Trying to free nonexistent resource <0000000007000000-0000000007ffffff>
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: mem region start 0x7000000 for 0x1000000 mapped at 0xd2380000
Mar 12 23:30:56 aida06 kernel: AIDAMEM: aidamem: driver assigned major number 253
Mar 12 23:31:13 aida06 kernel: xaida: open:
Mar 12 23:31:14 aida06 kernel: AIDAMEM: aidamem_open:
Mar 12 23:33:06 aida06 kernel: xaida: open:
Mar 13 05:37:30 aida06 syslogd 1.4.2: restart.
Mar 13 05:37:30 aida06 kernel: klogd 1.4.2, log source = /proc/kmsg started.
Mar 13 05:37:30 aida06 kernel: Using Xilinx Virtex440 machine description
Mar 13 05:37:30 aida06 kernel: Linux version 2.6.31 (nf@nnlxb.dl.ac.uk) (gcc version 4.2.2) #34 PREEMPT Tue Nov 15 15:57:04 GMT 2011
Mar 13 05:37:30 aida06 kernel: Zone PFN ranges: |
| Attachment 1: 210313_0138_Stats.png
|
|
| Attachment 2: 210313_0141_Temp.png
|
|
| Attachment 3: 210313_0142_Bias.png
|
|
| Attachment 4: 210313_0355_Stats.png
|
|
| Attachment 5: 210313_0356_Bias.png
|
|
| Attachment 6: 210313_0356_Temp.png
|
|
| Attachment 7: 210313_0528_Stats.png
|
|
| Attachment 8: 210313_0529_Temp.png
|
|
| Attachment 9: 210313_0530_Bias.png
|
|
| Attachment 10: 2210313_0640_AIDAissue.png
|
|
| Attachment 11: 210313_0647_aida6MMissing.png
|
|
|
|
198
|
Sun Mar 14 07:04:30 2021 |
OH | Sunday 14th March 2021 |
08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151
Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
It is also the least noisy FEE in the back DSSD
MERGE Data Link (20510): bad timestamp 10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
MERGE Data Link (20510): bad timestamp 10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262
09:01 System wide checks
Base Current Difference
aida07 fault 0x8c74 : 0x8c77 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0x2 : 0x5 : 3
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Statistics - attachment 1
Temperature - attachment 2
Bias and leakage current - attachment 3
11:18 System wide checks
Base Current Difference
aida07 fault 0x8c77 : 0x8c78 : 1
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Statistics - attachment 4
Temperatures - attachment 5
Bias and leakage - attachment 6
13.14 DAQ continues OK - file R56_285
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
13.18 System wide checks
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x8c77 : 0x8c78 : 1
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 27 8 3 4 2 2 2 2 3 4 10 : 54492
aida02 : 0 2 3 0 2 3 2 3 3 3 6 : 36416
aida03 : 18 9 1 2 0 4 1 3 3 3 6 : 36320
aida04 : 2 3 3 3 1 4 3 2 2 4 15 : 74224
aida05 : 26 72 72 54 53 49 22 18 7 4 7 : 72104
aida06 : 7 5 5 10 0 2 2 4 3 2 5 : 30932
aida07 : 2 4 2 1 1 3 2 3 2 3 7 : 39464
aida08 : 25 9 4 2 1 2 1 4 3 3 6 : 36716
aida09 : 4 11 3 5 2 4 1 2 3 3 6 : 36024
aida10 : 18 5 2 1 0 5 2 2 3 3 6 : 36144
aida11 : 25 10 4 1 1 4 2 3 2 3 6 : 35668
aida12 : 26 7 1 1 1 3 2 3 3 3 6 : 36496
13.19 FEE64 temperatures OK - attachment 7
Good events stats OK - attachment 8
Bias & leakage currents OK - attachment 9
Merger 4.5M items/s
Tape Server 14Mb/s
No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.
All histograms zero'd
16.00 DAQ contiues R56_361
System wide checks
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x8c77 : 0x8c79 : 2
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA Timestamp error counter test result: Passed 12, Failed 0
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 23 8 6 4 2 2 2 2 3 4 10 : 54524
aida02 : 1 3 4 2 1 2 1 4 3 3 6 : 36572
aida03 : 2 3 2 3 1 3 2 2 3 3 6 : 35936
aida04 : 19 7 2 3 1 4 3 3 1 4 15 : 73796
aida05 : 34 68 72 54 53 49 22 18 7 4 7 : 72104
aida06 : 11 10 4 10 0 2 2 4 3 2 5 : 30972
aida07 : 1 3 2 2 2 2 2 3 2 3 7 : 39420
aida08 : 24 10 5 2 1 3 1 4 3 3 6 : 36864
aida09 : 20 8 3 5 2 3 2 3 3 3 6 : 36704
aida10 : 2 3 2 3 1 3 3 2 3 3 6 : 36192
aida11 : 17 10 4 0 1 4 2 3 2 3 6 : 35604
aida12 : 20 6 3 2 1 3 2 2 2 4 6 : 37040
15.05 FEE64 temperatures OK - attachment 10
Good events stats OK - attachment 11
Bias & leakage currents OK - attachment 12
Merger 4.6M items/s
Tape Server 14Mb/s
No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.
Rate spectra - attachments 13 & 14
p+n junction HEC spectra - attachment 15
17.33 end of S452 run file R56_397
No Merger warning/error messages since last report. No MBS relay warning/error/messages since last restart.
BNC PB-5 config - attachment 16
CAEN 8xxx NIM bin config - attachment 17 |
| Attachment 1: 210314_0858_Stats.png
|
|
| Attachment 2: 210314_0859_Temp.png
|
|
| Attachment 3: 210314_0900_Bias.png
|
|
| Attachment 4: 210314_1116_Stats.png
|
|
| Attachment 5: 210314_1116_Temp.png
|
|
| Attachment 6: 210314_1117_Bias.png
|
|
| Attachment 7: Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi.png
|
|
| Attachment 8: Screenshot_2021-03-14_Statistics_aidas-gsi.png
|
|
| Attachment 9: 34.png
|
|
| Attachment 10: Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi(1).png
|
.png.png "Screenshot_2021-03-14_Temperature_and_status_scan_aidas-gsi(1).png")
|
| Attachment 11: Screenshot_2021-03-14_Statistics_aidas-gsi(1).png
|
.png.png "Screenshot_2021-03-14_Statistics_aidas-gsi(1).png")
|
| Attachment 12: 35.png
|
|
| Attachment 13: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi.png
|
|
| Attachment 14: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(1).png
|
.png.png "Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(1).png")
|
| Attachment 15: Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png
|
.png.png "Screenshot_2021-03-12_Spectrum_Browser_aidas-gsi(1).png")
|
| Attachment 16: 40.png
|
|
| Attachment 17: 41.png
|
|
| Attachment 18: 51.png
|
|
| Attachment 19: 52.png
|
|
| Attachment 20: Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png
|
.png.png "Screenshot_2021-03-14_Spectrum_Browser_aidas-gsi(3).png")
|
|
|
224
|
Thu Apr 15 22:38:18 2021 |
OH | Thursday 15th April - 16th April |
23:38 They are testing the finger detector
It is a lot of U they are putting into it which is passing through to AIDA ~3500Hz
Most seems to be punching through
23:45 Rate seems to be reduced to peaking at 1000Hz
00:06 DAQ Running to no storage
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xa
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse |
|
|
230
|
Fri Apr 16 11:42:45 2021 |
OH | Shifter checklist |
Every 30 minutes:
- Reload statistics (Web browser tab screen 1, workspace 2)
- Are all FEEs still showing a rate?
- If one shows 0 after multiple reloads Call Oscar or Tom.
- Options database monitor (Top middle terminal screen 2, workspace 2)
- Has the message changed since the last check?
- If so copy the message and place in elog
- Do the ucesb scalers make sense? (Web browser, screen 2, workspace 4)
- If one DSSD is reporting kHz of implants when the others are much less it is likely an ASIC getting stuck.
- Perform and ASIC check (ASIC control tab, web browser, screen 1, workspace 2)
- Ensure "Act on all FEE" and "Act on all asic" selected
- Click on drop down menu in the bottom of the screen and select check ASIC control
- It will take a minute or so be patient (MIDAS can only respond to one command at once so don't try to do anything else)
- A pop up will appear check that all options say ok
- Has the implant rate gone to a more normal amount?
- If not feel free to call
Every two hours:
- Take a screenshot of the statistics page (Web browser tab screen 1, workspace 2)
- Compare to the previous screenshot to check if anything has drastically changed
- Take a screenshot of the temperatures (Web browser tab screen 1, workspace 2)
- Do they all appear stable?
- Take a screenshot of the bias and leakage currents - (Top left terminal, screen 2, workspace 2)
- Perform the five system wide checks (Web browser tab screen 1, workspace 2)
- Make a note in the elog of which tests are successful
- If failed copy the text from the white box into the elog and make a note of it
- Enter the screenshots into the elog
- Update the leakage current spreadsheet (Web browser, screen 2, workspace 4) |
|
|
250
|
Mon Apr 19 23:01:02 2021 |
OH | 20th April 00:00-08:00 |
23:52 Noticed AIDA07 has dropped out of the merger and is producing no statistics
Powercycling to recover
00:14 All FEEs recovered from powercycle
aida09 running again at HEC threshold of 0x2 (Was set to 0x1 earlier in the day while testing the HEC event disparity in DSSD3)
Now writing to R34
Following the powercycle the rates in the FEEs are much improved and back to where they were before the 08:30 crash on the 19th. - Attachment 1
Waveforms also look improved - attachmens 2 and 3
Current system parameters
ASIC settings Dec19 (All LEC comparator 0xa)
Pulser at 2V and 2Hz
DSSD 1 and 2 Bias 160V
DSSD 3 at 120V (Will lower to 100 again during the beam off period tomorrow)
00:28 In analysis of R34_13 the number of HEC items in DSSD3 appears to make sense again. The number in aida09 are consistent with the number in aida05 and the same for aida11 and aida07.
This appears to confirm that the earlier HEC issues we were observing were hardware related but we are unsure what/why/how these issues were caused.
Analysis - attachment 4
00:40 Issue with one of the FRS magnets so they stop recording data. Leaving AIDA running.
01:08 While beam was down have reset DSSD3 bias to 100V as was agreed with Tom and Nic in email
Leakage current decreased as expected - attachment 5
Once beam is back will check HEC still as expected.
01:14 Last magnet in FRS has an issue with the water flowing. Either sensor issue or water flow is to low.
Hakke have increased the water flow to hopefully keep the water running.
Beam is back.
From ucesb it seems that the HEC events in DSSD3 are still appearing as expected. Will check a file once we have a complete file.
Checked R34_36 and the HEC events in DSSD3 are still consistent - attachment 6
2:20 Statistics ok - attachment 7
Temp - attachment 8
Bias and leakage current ok - attachment 9
All system wide checks ok except for WR error check:
Base Current Difference
aida07 fault 0x9154 : 0x915b : 7
aida10 fault 0x8d3 : 0x8d3 : 0
aida10 : WR status 0x10
White Rabbit error counter test result: Passed 10, Failed 2
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Note aida07 reports two bad timestamps in the merger
MERGE Data Link (3762): bad timestamp 6 3 0xc1817ea7 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56
MERGE Data Link (3762): bad timestamp 6 3 0xc1b77e25 0x04a8fc66 0x0000683974a8fc66 0x1677683974a8fc66 0x167768397a546a56
02:25 Remembered during a pre-experiment meeting with Vic, Patrick and Carl they mentioned the merger would track timestamp sequence errors
Can confirm this is true. Can see the two in aida07 in the counter in the merger stats page - attachment 10
02:45 Now up to 7 timestamp errors on aida07 in the merger
02:49 Ran R34_45 through AIDASort - It reports a greater number of implant events are able to have a front back match performed
Ratio between aida07 and 09 is 0.71 -> Double what it was before
Energy response of the HEC channels in aida07 and aida09 looks equivalent (left group aida09 right group aida11) - attahchment 11
04:40 Statistics ok - attachment 12
Temperatures ok - attachment 13
Bias and leakage currents - attachment 14
Clock status, ADC calibration and memory infomration all ok
aida07 fails both WR and FPGA
Base Current Difference
aida07 fault 0x915b : 0x9161 : 6
aida10 fault 0x8d3 : 0x8d3 : 0
aida10 : WR status 0x10
White Rabbit error counter test result: Passed 10, Failed 2
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
TS errors in the merger still only at 7 with all in aida07
06:25 Statistics - attachment 15
Temps ok - attachment 16
Bias and leakage currents ok - attachment 17
System wide checks: clock status, adc calibration and memory information all ok
WR multiple fails
Base Current Difference
aida05 fault 0xc9ca : 0xc9cb : 1
aida06 fault 0xe900 : 0xe901 : 1
aida07 fault 0x915b : 0x9166 : 11
aida08 fault 0x641e : 0x641f : 1
aida10 fault 0x8d3 : 0x8d3 : 0
aida10 : WR status 0x10
White Rabbit error counter test result: Passed 7, Failed 5
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA aida07 fails with difference of 1
No new TS sequence errors in merger
07:10 Beam has dropped out.
07:14 Beam is back |
| Attachment 1: 210420_0016_Stats.png
|
|
| Attachment 2: 210420_0021_Layout7.png
|
|
| Attachment 3: 210420_0022_Layout8.png
|
|
| Attachment 4: 210420_0026_R34_13_Analysis.txt
|
*** TDR format 3.3.0 analyser - TD - January 2019
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 255975543 ( 1356743.5 Hz)
Other data format: 5944457 ( 31507.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 32703 ( 173.3 Hz)
WR48-63: 32703 ( 173.3 Hz)
FEE64 disc: 1073243 ( 5688.5 Hz)
MBS info: 4805808 ( 25472.2 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 345437 ( 1830.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 188.669 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 46.534
2 0.000 63.789
3 0.000 54.847
4 0.000 174.519
5 0.000 123.945
6 0.000 113.898
7 0.000 135.923
8 0.000 59.774
9 0.000 27.296
10 0.000 12.263
11 0.000 65.598
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 14400003 100475 0 0 0 0 1767 1767 95813 1128 0 35602
1 23251868 63483 0 0 0 0 2988 2988 57507 0 0 21204
2 17192461 2025458 0 0 0 0 2415 2415 200396 1820232 0 34093
3 20019932 1908259 0 0 0 0 2815 2815 81941 1820688 0 32715
4 3059738 405674 0 0 0 0 403 403 100680 304188 0 35774
5 10595698 355900 0 0 0 0 1322 1322 58122 295134 0 21319
6 12037952 388178 0 0 0 0 1554 1554 105230 279840 0 34521
7 9390918 368608 0 0 0 0 1205 1205 81600 284598 0 31165
8 46854628 72459 0 0 0 0 5872 5872 60715 0 0 21758
9 43874737 60599 0 0 0 0 5426 5426 49747 0 0 17382
10 36184433 112868 0 0 0 0 4528 4528 103812 0 0 31745
11 19113175 82496 0 0 0 0 2408 2408 77680 0 0 28159
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 1641.051s ( 19.500 blocks/s, 1.219 Mb/s)
|
| Attachment 5: 210420_0109_Bias.png
|
|
| Attachment 6: 210420_0026_R34_36_Analysis.txt
|
*** TDR format 3.3.0 analyser - TD - January 2019
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 255999603 ( 1309621.8 Hz)
Other data format: 5920397 ( 30287.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 32728 ( 167.4 Hz)
WR48-63: 32728 ( 167.4 Hz)
FEE64 disc: 1044504 ( 5343.4 Hz)
MBS info: 4810437 ( 24608.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 327887 ( 1677.4 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 195.476 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 51.325
2 0.000 68.599
3 0.000 55.115
4 0.000 180.174
5 0.000 140.295
6 0.000 123.427
7 0.000 138.446
8 0.000 55.510
9 0.000 13.463
10 0.000 24.357
11 0.000 74.336
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 14611253 94841 0 0 0 0 1936 1936 89802 1167 0 33511
1 23423897 61346 0 0 0 0 3006 3006 55334 0 0 20222
2 17311487 2050104 0 0 0 0 2477 2477 195362 1849788 0 32021
3 20533060 1944396 0 0 0 0 2862 2862 77493 1861179 0 30741
4 2942569 381933 0 0 0 0 417 417 93930 287169 0 33667
5 9776355 338395 0 0 0 0 1239 1239 56110 279807 0 20096
6 11887130 369619 0 0 0 0 1555 1555 103862 262647 0 32503
7 9239609 348065 0 0 0 0 1200 1200 76985 268680 0 29470
8 49507336 72711 0 0 0 0 6129 6129 60453 0 0 21190
9 44170393 65100 0 0 0 0 5430 5430 54240 0 0 18016
10 32977762 112783 0 0 0 0 4022 4022 104739 0 0 30280
11 19618752 81104 0 0 0 0 2455 2455 76194 0 0 26170
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 5995.524s ( 5.337 blocks/s, 0.334 Mb/s)
|
| Attachment 7: 210420_0220_Stats.png
|
|
| Attachment 8: 210420_0229_Temp.png
|
|
| Attachment 9: 210420_0220_Bias.png
|
|
| Attachment 10: 210420_0225_MergerStats.png
|
|
| Attachment 11: Capture.PNG
|
|
| Attachment 12: 210420_0437_stats.png
|
|
| Attachment 13: 210420_0538_Temp.png
|
|
| Attachment 14: 210420_0439_Bias.png
|
|
| Attachment 15: 210420_0624_stats.png
|
|
| Attachment 16: 210420_0624_Temp.png
|
|
| Attachment 17: 210420_0625_Bias.png
|
|