 Thu Oct 20 01:30:10 2016, TD, Thursday 20 October 12x
|
09.30 DAQ found stopped, no merger error messages
Most recent file R63_291, last accessed 06.46
R63_291 ADC data synchronised
aidas1> ./analyser2 /TapeData/Sept2016/R63_291
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 29203
ADC data format: 237170622 ( 2006571.1 Hz)
Other data format: 1096655 ( 9278.2 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 113 ( 1.0 Hz)
RESUME: 113 ( 1.0 Hz)
SYNC100: 1077835 ( 9119.0 Hz)
FEE64 disc: 15061 ( 127.4 Hz)
MBS info: 3533 ( 29.9 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 175 ( 1.5 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 118.197 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.000 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 2.763 s
FEE module #: 6 elapsed dead time 0.275 s
FEE module #: 7 elapsed dead time 0.004 s
FEE module #: 8 elapsed dead time 0.003 s
FEE module #: 9 elapsed dead time 0.001 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.013 s
FEE module #: 15 elapsed dead time 0.028 s
FEE module #: 16 elapsed dead time 0.275 s
FEE module #: 17 elapsed dead time 6.010 s
FEE module #: 18 elapsed dead time 0.382 s
FEE module #: 19 elapsed dead time 0.555 s
FEE module #: 20 elapsed dead time 0.378 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.035 s
FEE module #: 23 elapsed dead time 0.168 s
FEE module #: 24 elapsed dead time 0.043 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 28.188s ( 1036.027 blocks/s, 64.752 Mb/s)
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks - SYNC errors & SYNC pulses fail - other checks OK - see attachments 4-7
AIDA SYNC stats - see attachment 8
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
09.43 DAQ starts (R64)
18.25 DAQ stop OK
Stop before 'scheduled' 18.30 DAQ stall
System wide checks - SYNC count & SYNC errors fail - other checks OK - see attachments 8-11
18.49 DAQ start (R65)
N.B. normal DAQ start - no DAQ RESET, SETUP, restart of merger & tapeserver etc
18.53 SYNC error counter baseline
19.21 DAQ found stalled
Most recent files R65_12 last access 19.17
Soft reset failed - nnaida4 connect timeouts - power cycle FEE64s
20.02 DAQQ start (R66) |
|
Fri Oct 21 01:41:54 2016, TD, Friday 21 October 14x
|
09.38 DAQ (R66) continues OK
Detector bias and leakage currents OK - see attachments 1&2
FEE64 temperatures OK - see attachment 3
System wide checks SYNC count & SYNC errors fail - see attachments 4-7 - other checks OK
R66_367 ADC data synchronised
AIDA SYNC stats - attachment 8
Some merger errors - see attachment 9
13.53 DAQ found stalled
Most recent files R66_396 last accessed 10.55
System wide checks SYNC count & SYNC errors fail - see attachments 10-13 - other checks OK
AIDA SYNC stats - attachment 14
No further merger error messages
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
14.02 DAQ starts (R67)
18.27 DAQ found stalled
Most recent file R67_121 last accessed 18.26
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
18.39 DAQ start (R68)
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
18.44 DAQ stalled
Most recent file R68_1 last accessed 18.44
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
18.53 DAQ starts (R70) |
|
Sat Oct 22 02:32:37 2016, TD, Saturday 22 October 15x
|
10.32 DAQ found stopped, no merger error messages
Most recent file R70_315, last accessed 06.46
R70_315 ADC data synchronised
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks - SYNC errors & SYNC pulses fail - other checks OK - see attachments 4-7
AIDA SYNC stats - see attachment 8
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.42 DAQ starts (R71)
18.25 DAQ stop OK
System wide test all OK *except* SYNC errors - see attachments 8-11
New SYNC error baseline
18.54 Repeat system wide checks whilst DAQ is still halted - SYNC errors *and* SYNC pulse now fail - see
attachments 12-15
This may suggest that the clock signal itself is affected by LN2 fill cycle
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
18.59 DAQ starts (R72)
nnaida15 drops data transfer #1 tag at DAQ GO
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
19.07 DAQ starts (R73)
19.17 DAQ stalled
Restart merger OK
DAQ STOP OK
Syten wide checks - SYNC error and SYNC pulse fail - dissimilar FEE64 counts in SYNC pulse
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
19.37 DAQ starts (R74)
|
|
Sun Oct 23 02:34:12 2016, TD, Sunday 23 October
|
10.34 DAQ found stopped, no merger error messages
Most recent file R74_296, last accessed 06.31
R70_315 ADC data synchronised
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
Some merger errors - see attachment 4
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.42 DAQ start (R75)
nnaida11 drops data transfer #1 tag at SAQ GO
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.46 DAQ start (R76)
18.18 DAQ found stopped
Most recent file R76_29 last accessed 11.52
Merger crashed - see attachment 5
aidas1> ./analyser /TapeData//Sept2016/R76_29
*** GREAT format 3.2.2 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 24384
ADC data format: 197754460 ( 1539809.2 Hz)
Other data format: 1194596 ( 9301.7 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 74 ( 0.6 Hz)
RESUME: 74 ( 0.6 Hz)
SYNC100: 1172671 ( 9131.0 Hz)
FEE64 disc: 18000 ( 140.2 Hz)
MBS info: 3777 ( 29.4 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 190 ( 1.5 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 128.428 s
FEE elapsed dead time(s)
1 0.000
2 0.007
3 0.000
4 0.000
5 0.674
6 2.583
7 0.000
8 0.000
9 0.000
10 0.000
11 0.000
12 0.000
13 0.000
14 0.000
15 0.276
16 0.124
17 0.435
18 0.050
19 0.622
20 0.009
21 0.000
22 0.000
23 0.352
24 0.000
25 0.000
26 0.000
27 0.000
28 0.000
29 0.000
30 0.000
31 0.000
32 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS
Other HEC Data
0 0 0 0 0 0 0 0 0 0
0 0
1 566974 49807 0 0 0 0 48983 824 0
0 62
2 5617753 50240 0 0 2 2 48984 1252 0
0 0
3 4298949 49615 0 0 0 0 48983 632 0
0 0
4 1980475 48985 0 0 0 0 48985 0 0
0 0
5 31381665 53750 0 0 12 12 48726 5000 0
0 0
6 35397222 52131 0 0 12 12 48004 326 3777
0 0
7 3679182 48987 0 0 0 0 48987 0 0
0 0
8 3893833 49613 0 0 0 0 48983 630 0
0 0
9 4909194 49313 0 0 0 0 48987 326 0
0 0
10 4099868 50957 0 0 0 0 48984 1973 0
0 0
11 1590517 49623 0 0 0 0 48987 636 0
0 0
12 2103665 48985 0 0 0 0 48985 0 0
0 0
13 3909540 50385 0 0 0 0 48990 1395 0
0 0
14 5011678 49344 0 0 0 0 48988 356 0
0 0
15 8074527 48904 0 0 10 10 48883 1 0
0 2
16 8822855 48949 0 0 5 5 48939 0 0
0 0
17 11581998 49241 0 0 10 10 48820 401 0
0 0
18 17158740 50227 0 0 1 1 48968 1257 0
0 0
19 14684270 49399 0 0 11 11 48747 630 0
0 0
20 6793305 47946 0 0 1 1 47944 0 0
0 0
21 4052765 50377 0 0 0 0 48986 1391 0
0 126
22 1773964 49339 0 0 0 0 48988 351 0
0 0
23 10790840 48872 0 0 10 10 48852 0 0
0 0
24 5580681 49607 0 0 0 0 48988 619 0
0 0
25 0 0 0 0 0 0 0 0 0
0 0
26 0 0 0 0 0 0 0 0 0
0 0
27 0 0 0 0 0 0 0 0 0
0 0
28 0 0 0 0 0 0 0 0 0
0 0
29 0 0 0 0 0 0 0 0 0
0 0
30 0 0 0 0 0 0 0 0 0
0 0
31 0 0 0 0 0 0 0 0 0
0 0
32 0 0 0 0 0 0 0 0 0
0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undef-d Samples
0 0 0 0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 3.586s ( 6799.896 blocks/s, 424.993 Mb/s)
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
18.50 DAQ starts (R77)
19.17 DAQ stalled due to disk housekeeping
Restart merger OK
DAQ STOP fails - nnaida1 timeout
Power cycle FEE64s
19.47 DAQ starts (R78) |
|
Sun Oct 23 10:22:36 2016, TD, Report - high - merger SEGV
|
Merger crashed during operation - see https://elog.ph.ed.ac.uk/AIDA/399 |
|
Mon Oct 24 02:46:57 2016, TD, Monday 24 October 19x
|
10.47 DAQ found stopped, no merger error messages
Most recent file R78_281, last accessed 06.46
R78_281 ADC data synchronised
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks - see attachments 4-11
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
11.07 DAQ starts (R79)
18.39 DAQ OK
System wide checks OK *except* SYNC errors & SYNC pulses - see attachments 12-15
18.47 DAQ stalls
Most recent file R79_182, last accessed 18.46
System wide checks OK *except* SYNC errors & SYNC pulses - see attachments 16-19
18.56 DAQ starts (R80) |
|
Tue Oct 25 02:06:34 2016, TD, Tuesday 25 October 13x
|
10.10 DAQ found stopped
Most recent file R81_169, last accessed 06.26
R81_169 ADC data synchronised
aidas1> ./analyser /TapeData/Sept2016/R81_169
*** TDR format 3.3.0 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 8369
ADC data format: 67712004 ( 1105598.7 Hz)
Other data format: 570667 ( 9317.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 17 ( 0.3 Hz)
RESUME: 17 ( 0.3 Hz)
SYNC100: 560442 ( 9150.9 Hz)
FEE64 disc: 8352 ( 136.4 Hz)
MBS info: 1839 ( 30.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 81 ( 1.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 61.245 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 0.000
2 0.000 0.000
3 0.000 0.000
4 0.000 0.000
5 0.244 0.254
6 0.181 0.191
7 0.000 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.002 0.000
16 0.000 0.000
17 0.001 0.000
18 0.002 0.000
19 0.003 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.085 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS
Other HEC Data
0 0 0 0 0 0 0 0 0 0
0 0
1 229453 23628 0 0 0 0 23359 269 0
0 22
2 2464367 23972 0 0 0 0 23362 610 0
0 0
3 1700233 23656 0 0 0 0 23358 298 0
0 0
4 650519 23359 0 0 0 0 23359 0 0
0 0
5 15435615 25773 0 0 4 4 23267 2498 0
0 0
6 15768889 25301 0 0 3 3 23291 165 1839
0 0
7 963142 23361 0 0 0 0 23361 0 0
0 0
8 773768 23661 0 0 0 0 23361 300 0
0 0
9 2244651 23523 0 0 0 0 23359 164 0
0 0
10 1833324 24095 0 0 0 0 23362 733 0
0 0
11 276357 23657 0 0 0 0 23362 295 0
0 0
12 542691 23359 0 0 0 0 23359 0 0
0 0
13 1695447 23996 0 0 0 0 23363 633 0
0 0
14 2181674 23530 0 0 0 0 23361 169 0
0 0
15 2203091 23362 0 0 1 1 23360 0 0
0 0
16 2179412 23361 0 0 0 0 23361 0 0
0 0
17 2317043 23549 0 0 1 1 23359 188 0
0 0
18 2579367 23969 0 0 1 1 23356 611 0
0 0
19 2589829 23664 0 0 3 3 23358 300 0
0 0
20 1296289 23357 0 0 0 0 23357 0 0
0 0
21 1624570 24024 0 0 0 0 23361 663 0
0 59
22 513131 23525 0 0 0 0 23360 165 0
0 0
23 3691921 23335 0 0 4 4 23327 0 0
0 0
24 1957221 23650 0 0 0 0 23359 291 0
0 0
25 0 0 0 0 0 0 0 0 0
0 0
26 0 0 0 0 0 0 0 0 0
0 0
27 0 0 0 0 0 0 0 0 0
0 0
28 0 0 0 0 0 0 0 0 0
0 0
29 0 0 0 0 0 0 0 0 0
0 0
30 0 0 0 0 0 0 0 0 0
0 0
31 0 0 0 0 0 0 0 0 0
0 0
32 0 0 0 0 0 0 0 0 0
0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0
*** Program elapsed time: 0.957s ( 8744.751 blocks/s, 546.547 Mb/s)
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 4-7
Merger error messages - see attachment 8
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.22 DAQ starts (R82)
16.07 DAQ continues OK
All system-wide checks OK
17.16 DAQ continues OK
All system-wide checks OK
18.27 All system wide checks OK *except* SYNC pulses - see attachment 9
18.34 DAQ stalled
Most recent file R82_125 last accessed 18.30
All system wide checks OK *except* SYNC errors and SYNC pulses - see attachments 10-13
18.47 AIDA -> safe state
FEE64 power OFF
DSSSD bias OFF |
|
Thu Oct 27 10:54:14 2016, TD, Thursday 27 October
|
14.20 DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
14.35 DAQ restarted for test purposes
18.55 DAQ stalled as expected
Most recent file R83_125 last accessed 18.30
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
20.48 DAQ starts (R86)
LEC/MEC fast comparator 0xff -> 0x20 |
|
Thu Oct 27 10:57:49 2016, TD, AIDA data readout
|
Developing real-time AIDA data synchronisation test using MIDASsort
To illustrate AIDA readout, ADC and discriminator data from a pulser event is plotted as a function
of the 100MHz AIDA internal timestamp (1us/channel)
Attachment 1 - nnaida8 number of ADC and discriminator data items
Attachment 2 - nnaida1-24 number of ADC and discriminator data items
Attachment 3 - 4*(FEE64 module # - 1) + ASIC # (ADC channels 0-95, disc channels 100-195) versus timestamp
(1us/channel)
The spectra clearly illustrate
- 500kHz ADC readout cycle
- discriminator readout precedes ADC readout by ~18us (this figure will be shaping time dependent)
- after-pulse due to pulser injection via simple charge terminator (no pole zero correction)
~84us after start of ADC readout (this figure will be pulser decay time and shaping time dependent)
Note events were identified (delimited) by time periods > 2us between successive ADC data items |
|
Fri Oct 28 06:12:50 2016, TD, Friday 28 October 12x
|
14.12 DAQ stalled as expected
Most recent file R86_245 last accessed 06.26
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
Some merger errors messages - all link 4 - see attachment 4
All system wide checks OK *except SYNC errors & SYNC pulses - see attachments 5-8
Restart merger
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
14.34 DAQ started (R87)
19.50 DAQ continues (R87)
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 9-12 |
|
Sat Oct 29 03:31:03 2016, TD, Saturday 29 October 11x
|
11.30 DAQ continues (R87)
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 1-4
11.36 DSSSD bias & leakage currents OK - see attachments 5-8
FEE64 temperatures OK - see attachment 9
16.46 DAQ continues (R87)
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 5-8
18.55 DAQ stalled
Most recent file R87_274 last accessed 18.47
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
19.10 DAQ starts (R89) |
|
Sat Oct 29 07:26:59 2016, TD, AIDA ADC & disc data synchronisation check 14x
|
Following analysis of AIDA ADC & disc readout ( see https://elog.ph.ed.ac.uk/AIDA/404 )
changed timestamp resolution from 1us/channel to 20.48us/channel
Offline sort (from disk file)
Attachments 1-3 s1-s32 # ADC data items per FEE64, s100-132 # disc data items per FEE64 (20.48us/channel)
Attachments 4-5 s1000 4 * ( module # - 1 ) + ASIC # versus timestamp (ch 0-95 ADC data, ch 100-195 disc data,
20.48us/channel)
Attachments 6-7 MIDASsort terminal session logs for *offline* sort
Online sort (from TapeServer)
Attachment 8 s1-s32 # ADC data items per FEE64, s100-132 # disc data items per FEE64 (20.48us/channel)
Attachments 9-10 s1000 4 * ( module # - 1 ) + ASIC # versus timestamp (ch 0-95 ADC data, ch 100-195 disc data,
20.48us/channel)
Attachments 11-12 MIDASsort terminal session logs for *online* sort
Attachment 13 MIDASsort FORTRAN sort routine sync.f
Attachment 14 Spectrum titles for the above sort routine
Note
1) data is discontinuous as a function of timestamp because we are sampling buffers
from the TapeServer rather than sorting all data as above
2) a low rate of timewarps is reported which appears to be an artifact of the buffer
sampling - no timewarps reported by analyses of the corresponding data files
|
|
Sun Oct 30 01:53:13 2016, TD, Sunday 30 October 9x
|
10.53 DAQ found stalled
Most recent files R89_102 last accessed 07.17
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
All system wide checks OK *except* Clock status, SYNC errors & SYNC pulses - see attachments 4-8 - unusual
messages/reports
One merger error messages - link 18 - see attachment 9
Restart merger
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
Standard soft reset did not work - one or more FEE64s dropped data transfer #1 link at DAQ GO
Retry - one, more FEE64s indicate error state - run control tab unresponsive
Power cycle FEE64s |
|
Sun Oct 30 03:26:55 2016, TD, Report - medium - cannot restore layout if currently displaying >16 galleries
|
Cannot restore layout if currently displaying >16 galleries
Obtain the error message 'loading Spectrum.tcl > can't read "dbtagx(31,1)": no such element in array' |
|
Sun Oct 30 04:37:31 2016, TD, Spectrum layouts created 8x
|
The following spectrum layouts have been created
1 All Rate spectra
2 All HitRate Spectra
3 p+n junction strip 1.8.L spectra
4 n+n ohmic strip 1.8.L spectra
5 p+n junction strip 1.8.H spectra
6 n+n ohmic strip 1.8.H spectra
5 p+n junction strip 1.8.W waveforms
6 n+n ohmic strip 1.8.W waveforms
The layouts can be found in the directory ~/LayOut and should be copied to /tmp
N.B Currently you cannot restore a layout to a spectrum display with > 16 galleries.
If spectrum display uses > 16 galleries, switch spectrum display to 16 galleries
or less before restoring a spectrum layout. |
|
Mon Oct 31 02:00:01 2016, TD, Monday 31 October 11x
|
11.00 DAQ continues OK
DAQ STOP OK
Most recent files R91_568 last accessed 10.59
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System wide checks OK *except* SYNC pulses & errors - see attachments 4-7
14.20 Per FEE64 screenshots p+n junction and n+n ohmic strip waveforms - see attachments 8-11
Preamplifier output noise looks OK for all FEE64s *except* nnaida5 & nnaida6 (which
have significantly higher ADC and disc rates than other FEE64s)
21.10 DAQ found stalled
Most recent file R94_99 last accessed 18.35
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
21.16 DAQ start (R96) |
|
Mon Oct 31 02:26:32 2016, TD, [How To] Start Runs (October 2016) 12x
|
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.
AIDA has multiple, independent data acquisition servers connected by an Ethernet network.
Response times to commands, information updates etc are ~10-60s - please be patient!
How To Start Run
1) Workspace 1 - Firefox web browser - Run Control tab
Select 'Update'
Check DAQ stopped (green 'GO' button displayed)
Check state of *all* data acquisition servers is 'stopped'
Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags
See attachment 1
2) Workspace 2 - Firefox web browser - TapeServer tab
Select 'Reload'
Check TapeServer stopped and disk file closed (green GO button displayed)
See attachment 2
3) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload'
Check Merge Program paused - see banner
Merge State = GOing : paused : xfer enabled : output paused
If not paused select 'Toggle Merge Program Pause State'
See attachment 3
4) Workspace 1 - Firefox web browser - ASIC Control tab
Select 'Reload'
Check 'Act on ALL FEE64 modules?' selected
Check 'Act on ALL ASICs' selected
Select 'Select required function' -> 'Check ASIC Control'
After ~60s a pop-up will appear showing the response from each ASIC of each data acquisition server.
Check all ASICs are OK and status = 0x001F0000.
Dismiss pop-up by selecting OK
See attachments 4-5
5) Workspace 1 - Firefox web browser - Run Control tab
Start DAQ - select green GO button
Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates
Check state of *all* data acquisition servers is 'going'
Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags
See attachment 6
6) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload' several times and check *all* enabled data links (0-23) are active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are inactive call for expert help and do NOT proceed!
Observe active spinner in MERGER server terminal window
See attachment 7
7) Workspace 2 - Firefox web browser - TapeServer tab
Check 'Volume Name' and 'File Name'
Start TapeServer and open disk file - select green 'GO' button
See attachment 8
8) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload' several times and check all enabled data links (0-23) are always active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are *always* inactive call for expert help and do NOT proceed!
Select 'Toggle Merge Program Pause State' to enable data merging
Select 'Reload' to observe merge rate (~ 9000 data items/s)
Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser
See attachment 9
9) Workspace 1 - Firefox web browser - Run Control tab
Select 'Start ASIC Data Readout'
See attachment 10
10) Workspace 2 - Firefox web browser - MERGE tab
Select Reload several times and check all enabled data links (0-23) are active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are *never* active call for expert help and do NOT proceed!
Select 'Reload' to observe merge rate (~ 0.5-2M data items/s)
See attachment 11
11) Workspace 2 - Firefox web browser - TapeServer tab
Select 'Reload' and observe number of blocks written to disk increasing
See attachment 12
12) Workspace 1 - Firefox web browser - Control tab
Select 'Save/Restore Settings'
From 'Save/Restore Module Settings' tab
select 'Reload'
select 'Module Names' whose settings should be saved (usually all)
select 'Generate new DB key' and make a note of the DB key generated
the DB key will have the format YYYYMMMDD-HH.MM.SS
The DB key will be used to name a directory to store AIDA settings
and is created uid: npg, gid: npgstaff, perms 755. However, its
contents are written uid: nfsnobody, gid nfsnobody. The default
directory (DB Key) permissions do not work - change the permissions
to 777 - before attempting to save data to this directory (DB Key).
launch a terminal and change the permissions of the DB key directory using the command
chmod 777 /MIDAS/DB/EXPERIMENTS/AIDA/[YYYYMMMDD-HH.MM.SS]
select 'Save Settings' to save AIDA ASIC, LED and waveform settings to
the directory (DB key)
11) Update BRIKEN Elog with the following information
AIDA
run filename (e.g. R99)
[# data blocks]
start date & time
[stop date & time]
ASIC settings DB key
additional comments
your name
Complete run sheet for written logbook |
|
Mon Oct 31 04:02:11 2016, TD, [How To] Stop Runs (October 2016)
|
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.
AIDA has multiple, independent data acquisition servers connected by an Ethernet network.
Response times to commands, information updates etc are ~10-60s - please be patient!
How to Stop Run
1) Workspace 2 - Firefox web browser - TapeServer tab
Select 'Reload'
Check number of blocks written to disk is increasing
See attachment 1
If TapeServer data rate is zero see https://elog.ph.ed.ac.uk/AIDA/303
2) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload'
Check merge rate (~0.5-2M data items/s)
See attachment 2
If MERGER data rate is zero see https://elog.ph.ed.ac.uk/AIDA/303
3) Workspace 1 - Firefox web browser - Run Control tab
Select 'Update'
Check state of *all* data acquisition servers is 'going'
Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags
Stop DAQ - select red 'STOP' button
Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates
Check state of *all* data acquisition servers is 'stopped'
Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags
See attachment 3
N.B. Occasionally the state of all data acquisition servers does not change to 'stopped'
and if this happens simply re-try by re-selecting 'STOP'
4) Workspace 2 - Firefox web browser - MERGE tab
Pause MERGE input - select 'Toggle Merge Program Pause State'
See attachment 4
5) Workspace 2 - Firefox web browser - TapeServer tab
Stop TapeServer and close disk file - select red 'STOP' button
See attachment 5
6) Complete BRIKEN Elog with the following information
AIDA
run filename (e.g. R99)
[# data blocks]
start date & time
[stop date & time]
ASIC settings DB key
additional comments
your name
Complete run sheet for written logbook |
|
Mon Oct 31 04:50:59 2016, TD, [How to] Monitor AIDA (October 2016)
|
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.
AIDA has multiple, independent data acquisition servers connected by an Ethernet network.
Response times to commands, information updates etc are ~10-60s - please be patient!
Every 4h
1) Desktop 7 - 2x PUTTY terminal sessions
Each PUTTY terminal session is connected to a CAEN N1419 4 ch HV Power Supply
Addr 00
Ch 0 = DSSSD # 1
:
:
Ch 3 = DSSSD # 4
Addr 01
Ch 0 = DSSSD # 5
Ch 1 = DSSSD # 6
Sequentially select each terminal and obtain a screenshot (Alt-PrtScn) of DSSSD #1 = #6 detector
bias voltages and leakage currents
Upload screenshots to BRIKEN Elog
If B2F/F11 ambient temperature information is available please add this information to the Elog
Compare DSSSD leakage currents (and temperatures) with previous observations
2) Desktop 1 - Firefox web browser - Temperatures tab
Select 'Reload'
Check temperatures
Virtex < 70 deg C
PSU < 45 deg C
ASIC < 70 deg C
Note that the ASIC temperatures for some data acquisition servers sometimes read back as 0.0
*If the temperature values above are exceeded switch off AIDA FEE64 electrical power* as follows
Desktop 6 - Firefox web browser
From 'Rly16 Control' select 'Switch ALL off'
and *now* seek expert help
*If* DAQ running also
3) Desktop 1 - Firefox web browser - Run Control tab
Select 'Update'
Check state of *all* data acquisition servers is 'going'
Check histogramming and data transfer #1 are enabled as indicated by 'H X' tags
5) Desktop 2 - Firefox web browser - MERGE tab
Select 'Reload'
Check merge rate (~0.5-2M data items/s) consistent with value observed at start of run
Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser
6) Desktop 2 - Firefox web browser - TapeServer tab
Select 'Reload'
Check data rate consistent with value observed at start of run |
|
Tue Nov 1 02:24:05 2016, TD, Tuesday 1 November 8x
|
11.24 DAQ found stalled
Most recent file R95_168 last accessed 06.31
R95_168 ADC & disc data synchronised - see attachment 8
spectrum s1000 - per ASIC data items as function of timestamp (20us/channel)
aidas1> ./analyser /TapeData/Sept2016/R95_168
*** TDR format 3.3.0 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 14568
ADC data format: 108169612 ( 1126585.0 Hz)
Other data format: 10690700 ( 111343.5 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 31 ( 0.3 Hz)
RESUME: 30 ( 0.3 Hz)
SYNC100: 877566 ( 9139.8 Hz)
FEE64 disc: 9810241 ( 102173.5 Hz)
MBS info: 2832 ( 29.5 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 2160 ( 22.5 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 96.015 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 0.000
2 0.000 0.000
3 0.000 0.000
4 0.000 0.000
5 1.994 2.016
6 1.408 1.439
7 0.000 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.015 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.045 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.099 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS
Other HEC Data
0 0 0 0 0 0 0 0 0 0
0 0
1 350112 43200 0 0 0 0 36623 6577 0
0 48
2 1247240 43778 0 0 0 0 36622 7156 0
0 0
3 2547061 39987 0 0 0 0 36626 3361 0
0 0
4 881007 40879 0 0 0 0 36623 4256 0
0 5
5 23965264 2536358 0 0 13 12 35831 2500502 0
0 38
6 23702370 1860427 0 0 11 11 36086 1821487 2832
0 844
7 1895595 40743 0 0 0 0 36623 4120 0
0 1
8 1603094 41634 0 0 0 0 36626 5008 0
0 325
9 3180182 1849501 0 0 0 0 36625 1812876 0
0 0
10 2576178 97708 0 0 0 0 36622 61086 0
0 0
11 447806 39812 0 0 0 0 36625 3187 0
0 0
12 905803 39001 0 0 0 0 36618 2383 0
0 0
13 2226595 147679 0 0 0 0 36624 111055 0
0 0
14 3021228 2593022 0 0 0 0 36626 2556396 0
0 513
15 4146089 40401 0 0 2 2 36617 3780 0
0 0
16 4264696 39555 0 0 0 0 36619 2936 0
0 0
17 2934508 416504 0 0 0 0 36623 379881 0
0 0
18 3319005 56583 0 0 0 0 36619 19964 0
0 0
19 6346836 41482 0 0 2 2 36604 4874 0
0 1
20 2392653 39249 0 0 0 0 36623 2626 0
0 283
21 2056421 41512 0 0 0 0 36622 4890 0
0 96
22 2824822 522339 0 0 0 0 36624 485715 0
0 5
23 7402741 39352 0 0 3 3 36590 2756 0
0 1
24 3932306 39994 0 0 0 0 36625 3369 0
0 0
25 0 0 0 0 0 0 0 0 0
0 0
26 0 0 0 0 0 0 0 0 0
0 0
27 0 0 0 0 0 0 0 0 0
0 0
28 0 0 0 0 0 0 0 0 0
0 0
29 0 0 0 0 0 0 0 0 0
0 0
30 0 0 0 0 0 0 0 0 0
0 0
31 0 0 0 0 0 0 0 0 0
0 0
32 0 0 0 0 0 0 0 0 0
0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0
*** Program elapsed time: 2.449s ( 5948.019 blocks/s, 371.751 Mb/s)
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System wide checks OK *except* SYNC pulses & errors - see attachments 4-7
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
11.46 Start DAQ (R96)
16.28 Stop DAQ OK
16.30 Start DAQ (R97)
BNC PB-4 pulser OFF (background run)
17.04 Stop DAQ
17.08 Ambient temperature +25.0 deg C, d.p. +8.4 deg C, RH 34.8%
17.11 Start DAQ (R98)
BNC PB-4 pulser ON
60Co cource (10-7023) placed on top of AIDA snout approximately above DSSSD #3
17.30 DAQ stalled due to excessive diak activity from data copying & analysis tasks
Power cycle required
18.09 DAQ start (R99)
18.38 DAQ stalled
Most recent file R99_7 last accessed 18.38
Power cycle rquired
19.42 DAQ start (R101)
BNC PB-4 pulser ON
60Co cource (10-7023) placed on top of AIDA snout approximately above DSSSD #3
(Phong's correction: 60Co source position was in front of AIDA snout toward upstream)
ASIC settings 2016Nov01-18.02.17 |