 Tue Oct 18 01:43:04 2016, TD, Tuesday 18 October 33x
|
09/42 DAQ found stopped, no merger error messages
Most recent file R52_211, last accessed 06.30
Restart merger OK
DAQ STOP OK
aidas1> ./analyser2 /TapeData/Sept2016/R52_211
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 7619
ADC data format: 61809254 ( 1624588.3 Hz)
Other data format: 354167 ( 9308.9 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 28 ( 0.7 Hz)
RESUME: 28 ( 0.7 Hz)
SYNC100: 347607 ( 9136.5 Hz)
FEE64 disc: 5364 ( 141.0 Hz)
MBS info: 1140 ( 30.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 88 ( 2.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 38.046 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.011 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.058 s
FEE module #: 6 elapsed dead time 0.165 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.000 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.002 s
FEE module #: 15 elapsed dead time 0.055 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.057 s
FEE module #: 18 elapsed dead time 0.089 s
FEE module #: 19 elapsed dead time 1.021 s
FEE module #: 20 elapsed dead time 0.000 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.000 s
FEE module #: 23 elapsed dead time 0.085 s
FEE module #: 24 elapsed dead time 0.004 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 1.793s ( 4249.377 blocks/s, 265.586 Mb/s)
R51_211 ADC data synchronised
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks - see attachments 4 & 5
SYNC errors and SYNC pulses fail, all other tests OK
Statistics - see attachments 6-16
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.02 DAQ start (R53)
18.20 DAQ OK
System-wide checks *all* OK
Statistics - see attachments 17-27
18.30 DAQ stalled on schedule
Most recent file R53_193 last accessed 18.30
AIDA SYNC rates zero in large fraction of FEE64s - see attachment 28
SYNC errors and SYNC pulses fail - see attachments 29-32
DAQ STOP
DAQ RESET
Restart merger
DAQ RESET OK
DAQ SETUP OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
System-wide checks *all* OK
18.45 DAQ starts (R54)
merging stops shortly after start of run - see attachment 33
DAQ HALT, RESET, SETUP OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
18.52 DAQ starts (R55) |
|
Mon Oct 17 04:10:30 2016, TD, Monday 17 October 7x
|
12.05 DAQ found stopped
Most recent file R49_226, last accessed 06.26
Restart merger OK
DAQ STOP OK
System-wide checks - see attachments 1-2
SYNC errors and SYNC pulses fail
ReSYNC OK
System-wide checks OK
Restart merger & tapeserver
DSSSD bias & leakage currents OK - see attachments 3 & 4
FEE64 temperatures OK - see attachment 5
DAQ start
#1 nnaida15 data transfer #1 tag drops at DAQ GO
disable H & data transfer #1
restart merger & tapeserver
enable H & data transfer #1
12.22 #2 DAQ start (R51) OK
20.33 DAQ found stopped
Most recent file R51_145, last accessed 18.30
Restart merger OK
DAQ STOP OK
System-wide checks - see attachments 6 & 7
SYNC errors and SYNC pulses fail
DAQ RESET & SETUP OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
20.41 DAQ GO (R52) |
|
Sun Oct 16 03:14:41 2016, TD, Sunday 16 October 16x
|
11.12 DAQ start (R47)
Detector bias and leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
Statistics - see attachments 4-14
11.56 Temperature +25.0 deg C, d.p. +8.4 deg C, RH 34.5%
20.33 DAQ found stopped
Most recent file R47_120 (last access 18.26)
R47_120 ADC data synchronised
Restart merger and request DAQ STOP - DAQ stops OK
System-wide tests SYNC errors and SYNC pulses - see attachments 15 & 16 - all other tests OK
DAQ RESET & SETUP OK
Normal restart which failed shortly after enabling ASIC readout - restarting merger did not
recover DAQ control - power cycle required
21.05 DAQ starts (R49)
|
|
Sat Oct 15 03:27:31 2016, TD, Saturday 15 October 10x
|
11.27 DAQ found stopped
Most recent file R42_164, last accessed 07.18
Restart merger OK
DAQ STOP OK
System-wide checks - see attachments 1-5
ASIC clock timestamp, SYNC errors fail *but* SYNC pulses OK
ReSYNC OK
System-wide checks OK
DSSSD bias & leakage currents OK - see attachments 6 & 7
FEE64 temperatures OK - see attachment 8
R42_164 ADC data synchronised
aidas1> ./analyser2 /TapeData/Sept2016/R42_164
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 2444
ADC data format: 19809141 ( 1404089.4 Hz)
Other data format: 131455 ( 9317.6 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 7 ( 0.5 Hz)
RESUME: 7 ( 0.5 Hz)
SYNC100: 129011 ( 9144.4 Hz)
FEE64 disc: 2010 ( 142.5 Hz)
MBS info: 420 ( 29.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 21 ( 1.5 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 14.108 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.000 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.334 s
FEE module #: 6 elapsed dead time 0.042 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.000 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.000 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.000 s
FEE module #: 18 elapsed dead time 0.005 s
FEE module #: 19 elapsed dead time 0.000 s
FEE module #: 20 elapsed dead time 0.000 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.000 s
FEE module #: 23 elapsed dead time 0.000 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 0.586s ( 4171.093 blocks/s, 260.693 Mb/s)
11.39 DAQ starts (R43)
12.07 Disk housekeeping caused TapeServer problem
DAQ stopped OK after some minutes and restarting merger
12.43 DAQ starts (R46)
18.26 DAQ stalled
Latest file R46_109 (most recent access 18.26)
Merger error messages - see attachment 9
Restart merger - error messages - see attachment 10
Analysis of R46_109
ADC data synchronised
aidas1> ./analyser2 /TapeData/Sept2016/R46_109
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 4688
ADC data format: 37985730 ( 1342216.8 Hz)
Other data format: 263662 ( 9316.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 25 ( 0.9 Hz)
RESUME: 25 ( 0.9 Hz)
SYNC100: 258788 ( 9144.2 Hz)
FEE64 disc: 3981 ( 140.7 Hz)
MBS info: 843 ( 29.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 41 ( 1.4 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 28.301 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.013 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.298 s
FEE module #: 6 elapsed dead time 0.200 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.001 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.003 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.086 s
FEE module #: 18 elapsed dead time 0.054 s
FEE module #: 19 elapsed dead time 0.097 s
FEE module #: 20 elapsed dead time 0.000 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.001 s
FEE module #: 23 elapsed dead time 0.000 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 2.102s ( 2230.721 blocks/s, 139.420 Mb/s)
11.00 Sunday 16 October
DAQ STOP requested - nnaida1-24 all stop *except* nnaida6 which remains going - Run Control displays RESET
only
DAQ RESET
System-wide checks all OK
Normal setup and DAQ restart (see https://elog.ph.ed.ac.uk/AIDA/388 )
DAQ SETUP
|
|
Thu Oct 13 16:12:06 2016, TD & CG, Friday 14 October 14x
|
DAQ continues (R33)
aidas1> ./analyser2 /TapeData/Sept2016/R33_0
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 258854184 ( 1075951.6 Hz)
Other data format: 2233816 ( 9285.1 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 95 ( 0.4 Hz)
RESUME: 95 ( 0.4 Hz)
SYNC100: 2200965 ( 9148.5 Hz)
FEE64 disc: 26965 ( 112.1 Hz)
MBS info: 5696 ( 23.7 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 273 ( 1.1 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 240.582 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.026 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.278 s
FEE module #: 6 elapsed dead time 1.219 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.000 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.001 s
FEE module #: 15 elapsed dead time 0.014 s
FEE module #: 16 elapsed dead time 0.015 s
FEE module #: 17 elapsed dead time 0.703 s
FEE module #: 18 elapsed dead time 0.196 s
FEE module #: 19 elapsed dead time 1.037 s
FEE module #: 20 elapsed dead time 0.063 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.006 s
FEE module #: 23 elapsed dead time 0.073 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 22.278s ( 1436.394 blocks/s, 89.775 Mb/s)
00.16 DSSSD bias and leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
Temperature +25.0 deg C, d.p. +8.2 deg C, RH 33.8%
00.25 DAQ stopped
Unable to restart merger - see attachment 4 - DAQ STOP OK
System wide tests fail - see attachments 5 & 6
DAQ RESET & SETUP, ReSYNC OK
System wide checks fail - as above
Several FEE64 power cycles and eventually an aidas1 reboot due to
1) timeouts from random FEE64s following power cycle + 10 mins
2) data transfer links (X) dropping at DAQ GO
01.49 DAQ starts (R40)
08.31 DAQ found stopped
Most recently accessed file R40_91 @ 06.26
Restart merger & request DAQ STOP
Transient timeout from nnaida8 but STOP completed OK
11.10 DSSSD bias and leakage currents OK - see attachments 7 & 8
FEE64 temperatures OK - see attachment 9
System-wide checks - system clock, SYNC errors and SUNC count fail - see attachments 10-13
ReSYNC fails - see attachment 14
11.30 Analysis of R40 data
R40_91 ADC data synchronised
aidas1> ./analyser2 /TapeData/Sept2016/R40_91
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 30880
ADC data format: 249997809 ( 1194334.5 Hz)
Other data format: 1952111 ( 9326.0 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 55 ( 0.3 Hz)
RESUME: 55 ( 0.3 Hz)
SYNC100: 1915685 ( 9152.0 Hz)
FEE64 disc: 30044 ( 143.5 Hz)
MBS info: 6272 ( 30.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 303 ( 1.4 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 209.320 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.000 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.167 s
FEE module #: 6 elapsed dead time 0.529 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.000 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.002 s
FEE module #: 15 elapsed dead time 0.018 s
FEE module #: 16 elapsed dead time 0.043 s
FEE module #: 17 elapsed dead time 0.098 s
FEE module #: 18 elapsed dead time 0.035 s
FEE module #: 19 elapsed dead time 0.378 s
FEE module #: 20 elapsed dead time 0.010 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.000 s
FEE module #: 23 elapsed dead time 0.046 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 6.613s ( 4669.392 blocks/s, 291.837 Mb/s)
21.52 DAQ starts (Sept2016/R41) BRIKEN filename 252CfwithAida75mm
Correlation scaler reset requested by BRIKEN DAQ following start of run to disk
End of AIDA snout positioned at edge of (CH2)n moderator aperture for
ORNL HPGe clover detectors
RIKEN 252Cf source ~25k neutrons/s
FEE64 temperatures OK
*All* system-wide checks OK
B2F/F11 temperature + deg C, d.p. + deg C,RH %
ASIC settings 2016Sep02-18.17.01
slow comparator 0x10 -> 0x0a
LEC/MEC fast comparators # 1, 55 & 58 disabled for all FEE64s
Master MACB serial #13 (see https://elog.ph.ed.ac.uk/AIDA/367)
Master MACB mode 5
TS_SYNC_PHASE 0
22.17 DAQ stops OK
BRIKEN DAQ stopped at about same time
22.45 DAQ starts (R42)
AIDA withdrawn from (CH2)n moderator, 252Cf source removed |
|
Thu Oct 13 06:07:43 2016, TD & CG, Thursday 13 October
|
11.40 Power cycle AIDA FEE64s
Set TS_SYNC_PHASE 0
DSSSD bias/leakage current OK - see attachment 1-2
FEE64 temperatures OK - see attachment 3
*All* system-wide checks OK
B2F/F11 temperature + deg C, d.p. + deg C,RH %
ASIC settings 2016Sep02-18.17.01
slow comparator 0x10 -> 0x0a
LEC/MEC fast comparators # 1, 55 & 58 disabled for all FEE64s
Master MACB serial #13 (see https://elog.ph.ed.ac.uk/AIDA/367)
Master MACB mode 5
12.15 DAQ starts (file Sept2016/R24)
15.16 DAQ stops OK
15.18 DAQ starts (R25)
Pulser walkthrough
BNC PB-4 amplitudes 10,000 - 90,000 @ 10,000 step
See attachment 4
Spectra saved 13.10.16 15.32 (08.32 FEE64s)
15.30 DAQ stops OK
15.38 DAQ starts (R26)
Pulser walkthrough
BNC PB-4 amplitudes 8,000 - 2,000 @ 2,000 step
See attachment 5
Spectra saved 13.10.16 15.47 (08.47 FEE64s)
15.45 DAQ stops
15.50 DAQ starts (R27)
LEC/MEC FSR 20MeV -> 1GeV
BNC PB-4 amplitude 90,000
Spectra saved 13.10.16 16.07 (09.05 FEE64s)
16.05 DAQ stops
16.12 DAQ starts (R28)
LEC/MEC FSR 1GeV -> 20MeV
BNC PB-4 off (noise + background activity only)
Spectra saved 13.10.16 16:49 (saved 09.49 FEE64s)
16.48 DAQ stops
16.57 DAQ starts (R29)
BNC PB-4 90,000 amplitude ~23Hz freq.
17.26 Lost SYNC pulses. Merger+TapeServer stopped writing to file (noticed 18.20)
18.25 R29 closed.
19:20 NIM bin supplying TTL clock to master MACB failed at or around the same time as the DAQ stop occurred.
Need to replace NIM bin before clock can be supplied.
21.20 DAQ starts (R32)
23.51 DAQ found stopped, most recent file R32_26 (last access 22.34 13.10.16)
23.56 Restarted merger & immediately requested DAQ STOP - OK
DAQ reset & setup
System-wide tests all OK
00.01 DAQ starts (R33) |
|
Wed Oct 12 02:58:19 2016, TD & CG, Wednesday 12 October
|
10.50 Merger & TapeServer found stopped
connect errors - see attachment 1
DSSSD bias and leakage currents OK - see attachments 2 & 3
Most recent data file R17_440 last accessed 08.22 12.10.16
R17_440 ADC data synchronised
aidas1> ./analyser2 /TapeData/Sept2016/R17_440
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 30546
ADC data format: 147630891 ( 1189229.1 Hz)
Other data format: 101593923 ( 818381.9 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 101 ( 0.8 Hz)
RESUME: 101 ( 0.8 Hz)
SYNC100: 1133804 ( 9133.3 Hz)
FEE64 disc: 100456290 ( 809217.8 Hz)
MBS info: 3627 ( 29.2 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 151 ( 1.2 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 124.140 s
FEE module #: 1 elapsed dead time 0.728 s
FEE module #: 2 elapsed dead time 0.184 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 1.034 s
FEE module #: 6 elapsed dead time 3.462 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.064 s
FEE module #: 10 elapsed dead time 0.003 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.062 s
FEE module #: 15 elapsed dead time 0.018 s
FEE module #: 16 elapsed dead time 0.005 s
FEE module #: 17 elapsed dead time 0.817 s
FEE module #: 18 elapsed dead time 0.035 s
FEE module #: 19 elapsed dead time 0.048 s
FEE module #: 20 elapsed dead time 0.000 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.103 s
FEE module #: 23 elapsed dead time 0.216 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 14.129s ( 2161.951 blocks/s, 135.122 Mb/s)
Conclude DAQ ran without problem for c. 16.5h which is a significant improvement
assuming that it can be consistently reproduced.
11.20 Merger restart did not clear connect/timeout issues
Re-route BNC - BNC RG58 cable with 50MHz clock from BRIKEN to
AIDA Master MACB external clock input (rear panel) so that it will
not be stepped upon.
Power cycle AIDA FEE64s
N.B. I selected 'Switch ALL On' rather than 'Sequence ALL On' by mistake and one
of the USB ac mains controller output 250VAC/16A fuses blew - use 'Sequence ALL On'
TS_SYNC_PHASE 0
FEE64 temperatures OK
*All* system-wide checks OK
B2F/F11 temperature +24.6 deg C, d.p. +8.9 deg C,RH 36.6%
ASIC settings 2016Sep02-18.17.01
slow comparator 0x10 -> 0x0a
LEC/MEC fast comparators # 1, 55 & 58 disabled for all FEE64s
Master MACB serial #13 (see https://elog.ph.ed.ac.uk/AIDA/367)
Master MACB mode 5
14.05 DAQ starts (file Sept2016/R21)
15.30 Sequence of three electrical power dips over 30 minute period. RIKEN public address
system reports that TEPCO (Tokyo Electric Power Company) - the site supplier - is
having region-wide problems.
The third power dip stopped DAQ.
Unable to recover by restarting merger
Power cycle FEE64s
16.20 DAQ restarts (file Sept2016/R22)
19.05 DAQ stops
Because of risk of power outages due to TEPCO distribution problems we have powered
down AIDA and Julabo FL1006 chiller and disconnected from ac mains. |
|
Tue Oct 11 03:34:02 2016, CG & TD, Tuesday 11 October 16x
|
10.40 DAQ found stopped
All FEE64 AIDA SYNC rates zero, AIDA SYNC count ~ 13.8M => DAQ stopped @ 06.35
System wide checks - see attachments 1-4 - all other system-wide checks OK
11.15 Replace cable for Master MACB external clock signal
- change from c. 4m RG174 (Lemo-00 to Lemo-00) to c. 4m RG58 (BNC to BNC)
Power cycle AIDA FEE64s
Set TS_SYNC_PHASE 0
DSSSD bias/leakage current OK - see attacment 5-6
FEE64 temperatures OK - see attachment 7
*All* system-wide checks OK (N.B. several ReSYNCs required)
B2F/F11 temperature + deg C, d.p. + deg C,RH %
ASIC settings 2016Sep02-18.17.01
slow comparator 0x10 -> 0x0a
Master MACB serial #13 (see https://elog.ph.ed.ac.uk/AIDA/367)
Master MACB mode 5
11.58 DAQ starts
15.41 DAQ OK
15.44 DAQ stop requested
15.46 DAQ start
Data transfer to disk enabled to test DAQ correlation real-time monitor
22.50 AIDA SYNCs OK
# Master SYNCs *not equal* to # FEE64 SYNCs - all FEE64 SYNCs # equal
- all other system-wide checks OK - see attachments 8-15
Merger and TapeServer OK
R17_197 ADC data synchonised
aidas1> ./analyser2 /TapeData/Sept2016/R17_197
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 165074646 ( 1182230.4 Hz)
Other data format: 96013354 ( 687627.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 113 ( 0.8 Hz)
RESUME: 113 ( 0.8 Hz)
SYNC100: 1275326 ( 9133.6 Hz)
FEE64 disc: 94733738 ( 678463.4 Hz)
MBS info: 4064 ( 29.1 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 191 ( 1.4 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 139.630 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.206 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 1.084 s
FEE module #: 6 elapsed dead time 4.015 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.201 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.003 s
FEE module #: 14 elapsed dead time 0.193 s
FEE module #: 15 elapsed dead time 0.024 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 1.198 s
FEE module #: 18 elapsed dead time 0.040 s
FEE module #: 19 elapsed dead time 0.124 s
FEE module #: 20 elapsed dead time 0.000 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.187 s
FEE module #: 23 elapsed dead time 0.258 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 19.555s ( 1636.436 blocks/s, 102.277 Mb/s) |
|
Mon Oct 10 03:15:59 2016, CG & TD, Monday 10 October 25x
|
10.30 DAQ found stopped
All FEE64 AIDA SYNC rates zero, AIDA SYNC count ~ 4.4M => DAQ stopped @ 01.40
3He detector DAQ off - no 50MHz clock to Master MACB
System wide checks - see attachments 26-18
N.B. It would appear that the DAQ stalled (TS_SYNC_PHASE 1) c. 01.40 and at some
later time (c. 06.00) the 3He detector DAQ was powered off which means that AIDA
lost the external 50MHz clock to the Master MACB - system-wide screenshots should
be interpreted with caution.
14.15 3He detector DAQ 50MHz clock restarted
VME and NIM crates ON
Excecute command ConfClock_50MHz on 3He detector DAQ computer
Power cycle AIDA FEE64s
Set TS_SYNC_PHASE 2
DSSSD bias/leakage current OK
FEE64 temperatures OK
*All* system-wide checks OK
B2F/F11 temperature + deg C, d.p. + deg C,RH %
ASIC settings 2016Sep02-18.17.01
slow comparator 0x10 -> 0x0a
Master MACB serial #13 (see https://elog.ph.ed.ac.uk/AIDA/367)
Master MACB mode 5
14.20 DAQ starts (no data transfer)
18.10 DAQ found stopped
All FEE64 AIDA SYNC rates zero, AIDA SYNC count ~ 3.4M => DAQ stopped @ 16.50
System-wide checks - see attachments 16-9
Power cycle AIDA FEE64s
Set TS_SYNC_PHASE 3
ASIC settings 2016Sep02-18.17.01
slow comparator 0x10 -> 0x0a
20.10 DAQ found stalled
System-wide checks - see attachments 8-1
Power cycle AIDA FEE64s
Set TS_SYNC_PHASE 0
ASIC settings 2016Sep02-18.17.01
slow comparator 0x10 -> 0x0a
20.35 DAQ starts (no data transfer)
slow comparator 0x10 -> 0x0a
18.35 DAQ starts (no data transfer)
20.16 DAQ found stopped
All FEE64 AIDA SYNC rates zero, AIDA SYNC count ~ 120k (17/24 FEE64s) or 1.49M (7/24) => FEE64s stopped @ 18.40 & 19.40
20.36 DAQ starts (no data transfer)
23.47 DAQ OK |
|
Sun Oct 9 11:47:34 2016, CG, TD, Sunday 9th October 9x
|
12.30 Following scheduled power outages of October 8
power up AIDA and Julabo FL11006 recirculating chiller
Julabo FL11006 set point +20 deg C
Temperature +25.3 deg C, d.p. +10.6 deg C, RH 39.3%
boot aidas1
aidas1 filesystem checks (fsck) run from c. 12.30 to 15.58
System wide checks OK *except* nnaida21 failed ADC calibration
16.55 DAQ starts (no data transfer)
ASIC settings 2016Sep02-18.17.01
slow comparator threshold 0x10 -> 0x0a
TS_SYNC_PHASE 0
DSSSD bias & leakage current OK - see attachments 8 & 9
FEE64 temperatures OK - see attachment 7
17.30 DAQ stops
DSSSD bias off
AIDA withdrawn from (CH2)n moderator
DSSSD bias on
17.40 DAQ starts
19.15 Noticed DAQ stopped producing SYNC pulses at some point.
A reload of the run control tab sees them all as going still, but only 20% are reporting any SYNC output in the stats page.
All FEEs have failed the SYNC errors since baseline check and the values reported by the SYNC pulses received tests vary
by +- 3 of a central value.
See attachments 1-6.
The merger and tape server were not set up before this run and are not currently running.
TS_SYNC phase changed from 0 -> 1 and ReSYNC
22.20 DAQ OK
AIDA SYNCs OK
All system-wide checks OK *except* nnaida21 fails ADC calibration (no change) |
|
Fri Oct 7 02:43:47 2016, TD & CG, Analysis of Sept2016/R13_47
|
Analysis of R13_47 - the data file that was being written when DAQ stalled
yesterday evening - see https://elog.ph.ed.ac.uk/AIDA/371
aidas1> ./analyser2 /TapeData/Sept2016/R13_46
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 145759250 ( 544585.9 Hz)
Other data format: 115328750 ( 430891.4 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 74 ( 0.3 Hz)
RESUME: 74 ( 0.3 Hz)
SYNC100: 2449126 ( 9150.4 Hz)
FEE64 disc: 112871466 ( 421710.5 Hz)
MBS info: 8010 ( 29.9 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 382 ( 1.4 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 267.652 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.824 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.275 s
FEE module #: 6 elapsed dead time 0.488 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.017 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.023 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.012 s
FEE module #: 18 elapsed dead time 0.000 s
FEE module #: 19 elapsed dead time 0.912 s
FEE module #: 20 elapsed dead time 0.000 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.000 s
FEE module #: 23 elapsed dead time 0.000 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 17.016s ( 1880.624 blocks/s, 117.539 Mb/s)
aidas1> ./analyser2 /TapeData/Sept2016/R13_47
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 11864
ADC data format: 54019518 ( 544839.2 Hz)
Other data format: 42776197 ( 431439.3 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 29 ( 0.3 Hz)
RESUME: 29 ( 0.3 Hz)
SYNC100: 907301 ( 9151.0 Hz)
FEE64 disc: 41865866 ( 422257.8 Hz)
MBS info: 2972 ( 30.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 135 ( 1.4 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 99.148 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.179 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.199 s
FEE module #: 6 elapsed dead time 0.100 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.005 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.003 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.021 s
FEE module #: 18 elapsed dead time 0.000 s
FEE module #: 19 elapsed dead time 0.311 s
FEE module #: 20 elapsed dead time 0.000 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.000 s
FEE module #: 23 elapsed dead time 0.000 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 6.238s ( 1901.806 blocks/s, 118.863 Mb/s)
Final SYNCs of R13_47
aidas1. ./analyser2 v /TapeData/Sept2016/R13_47 | grep SYNC
:
:
*** SYNC100 timestamp: block: 11864 ptr: 2151 data: 0x81401340 module: 1 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2153 data: 0x82401340 module: 2 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2155 data: 0x83401340 module: 3 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2157 data: 0x84401340 module: 4 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2159 data: 0x85401340 module: 5 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2161 data: 0x86401340 module: 6 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2163 data: 0x87401340 module: 7 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2165 data: 0x88401340 module: 8 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2167 data: 0x89401340 module: 9 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2169 data: 0x8A401340 module: 10 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2171 data: 0x8B401340 module: 11 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2173 data: 0x8C401340 module: 12 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2175 data: 0x8D401340 module: 13 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2177 data: 0x8E401340 module: 14 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2179 data: 0x8F401340 module: 15 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2181 data: 0x90401340 module: 16 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2183 data: 0x91401340 module: 17 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2185 data: 0x92401340 module: 18 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2187 data: 0x93401340 module: 19 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2189 data: 0x94401340 module: 20 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2191 data: 0x95401340 module: 21 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2193 data: 0x96401340 module: 22 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2195 data: 0x97401340 module: 23 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 2197 data: 0x98401340 module: 24 information type: 4 information
field: 0x00001340 ts: 0x000001340B4800A0
*** SYNC100 timestamp: block: 11864 ptr: 6807 data: 0x81401340 module: 1 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6809 data: 0x82401340 module: 2 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6811 data: 0x83401340 module: 3 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6813 data: 0x84401340 module: 4 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6815 data: 0x85401340 module: 5 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6817 data: 0x86401340 module: 6 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6819 data: 0x87401340 module: 7 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6821 data: 0x88401340 module: 8 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6823 data: 0x89401340 module: 9 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6825 data: 0x8A401340 module: 10 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6827 data: 0x8B401340 module: 11 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6829 data: 0x8C401340 module: 12 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6831 data: 0x8D401340 module: 13 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6833 data: 0x8E401340 module: 14 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6835 data: 0x8F401340 module: 15 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6837 data: 0x90401340 module: 16 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6839 data: 0x91401340 module: 17 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6841 data: 0x92401340 module: 18 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6843 data: 0x93401340 module: 19 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6845 data: 0x94401340 module: 20 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6847 data: 0x95401340 module: 21 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6849 data: 0x96401340 module: 22 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6851 data: 0x97401340 module: 23 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
*** SYNC100 timestamp: block: 11864 ptr: 6853 data: 0x98401340 module: 24 information type: 4 information
field: 0x00001340 ts: 0x000001340B4C00A0
Analysis of the end of file R13_47 - see attachment 1
Offline analysis of R13_3 and R13_47 shows AIDA ADC data synchronised |
|
Fri Oct 7 02:29:19 2016, TD & CG, Friday 7 October 22x
|
10.29 DAQ left running overnight
Histograms only - no data transfer to Merger & TapeServer
DSSSD biases & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
DAQ run control OK - see attachment 4
Statistics - see attachments 5-16
N.B. some FEE64s are producing SYNCs, some not, in particular nnaida6 is not
System-wide tests - see attachments 17-22
11.31 DAQ recovered with STOP-GO (no data transfer)
AIDA SYNCs OK for all FEE64s
22.46 DAQ still running OK
Power down in preparation for tomorrow's planned site-wide power outage |
|
Thu Oct 6 13:22:24 2016, TD, Images of setup at F11 - 4
|
|
|
Thu Oct 6 13:20:34 2016, TD, Images of setup at F11 - 3
|
|
|
Thu Oct 6 13:09:50 2016, TD, Images of setup at F11 - 2
|
|
|
Thu Oct 6 12:39:14 2016, TD, Images of setup at F11
|
|
|
Thu Oct 6 11:16:40 2016, CG & TD, Thursday 6 October 9x
|
DSSSD bias voltages and leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
Install MACB serial #13 (firmware 16.4.15) as Master MACB, mode 5
System-wide checks all OK
N.B. nnaida19 ASIC clock OK (cf previous Elog entries) - see attachment 4
Setup AIDA-BELEN DAQ correlation - RIBF still to do - see attachments 5 & 6
reset request and trigger signals to Master MACB - NIM, width ~200ns
BELEN clock (10Hz) triggers BELEN BNC PB-5 and AIDA Master MACB
*** MBS information: block: 9349 ptr: 3725 data: 0x86807FF5 module: 6 information type: 8 information index: 0 MBS data:7FF5 ts: 0x00000026D0FE47A2
*** MBS information: block: 9349 ptr: 3727 data: 0x8681AF75 module: 6 information type: 8 information index: 1 MBS data:AF75 ts: 0x00000026D0FE47A2
*** MBS information: block: 9349 ptr: 3729 data: 0x86820000 module: 6 information type: 8 information index: 2 MBS data:0000 ts: 0x00000026D0FE47A2
*** MBS information: block: 9362 ptr: 7641 data: 0x8680A5AD module: 6 information type: 8 information index: 0 MBS data:A5AD ts: 0x00000026D196DE82
*** MBS information: block: 9362 ptr: 7643 data: 0x8681AF9B module: 6 information type: 8 information index: 1 MBS data:AF9B ts: 0x00000026D196DE82
*** MBS information: block: 9362 ptr: 7645 data: 0x86820000 module: 6 information type: 8 information index: 2 MBS data:0000 ts: 0x00000026D196DE82
0x0000AF9BA5AD - 0x0000AF757FF5 = 0x2625B8 = 2500024 * 40ns = 100000960ns = 0.1s
0x26D196DE82 - 0x26D0FE47A2 = 0x9896E0 = 10000096 * 10ns = 100000960ns = 0.1s
BELEN DAQ manual reset -> Master MACB reset request
*** MBS information: block: 9375 ptr: 3043 data: 0x86808A8A module: 6 information type: 8 information index: 0 MBS data:8A8A ts: 0x00000026D22F755E
*** MBS information: block: 9375 ptr: 3045 data: 0x8681001F module: 6 information type: 8 information index: 1 MBS data:001F ts: 0x00000026D22F755E
*** MBS information: block: 9375 ptr: 3047 data: 0x86820000 module: 6 information type: 8 information index: 2 MBS data:0000 ts: 0x00000026D22F755E
0x0000001F8A8A = 2067082 * 40ns = 82683280ns = 82.7us (which is < 0.1s as expected)
*** MBS information: block: 9388 ptr: 3291 data: 0x8680B041 module: 6 information type: 8 information index: 0 MBS data:B041 ts: 0x00000026D2C80C3A
*** MBS information: block: 9388 ptr: 3293 data: 0x86810045 module: 6 information type: 8 information index: 1 MBS data:0045 ts: 0x00000026D2C80C3A
*** MBS information: block: 9388 ptr: 3295 data: 0x86820000 module: 6 information type: 8 information index: 2 MBS data:0000 ts: 0x00000026D2C80C3A
*** MBS information: block: 9401 ptr: 5841 data: 0x8680D5F9 module: 6 information type: 8 information index: 0 MBS data:D5F9 ts: 0x00000026D360A31A
*** MBS information: block: 9401 ptr: 5843 data: 0x8681006B module: 6 information type: 8 information index: 1 MBS data:006B ts: 0x00000026D360A31A
*** MBS information: block: 9401 ptr: 5845 data: 0x86820000 module: 6 information type: 8 information index: 2 MBS data:0000 ts: 0x00000026D360A31A
19.05 R13_0 starts
23.19 Returned from dinner to find DAQ had stopped writing to disk at 22.19. Merger paused and Tape run ended successfully.
No SYNC pulses being issued.
Carried out system-wide checks. Clock status, ADC calib and ASIC clock passed for all 24 FEEs.
Attachments 7 & 8 show errors messages issued upon fail of SYNC errors and SYNC pulses system checks.
From attachment 8
( 6016000 SYNC / 9150 SYNC/s ) * 24 ~ 15800s => DAQ stopped 23.07 which is not consistent with observation
as final write to disk files was c. 22.19 - note 23.21 for R13_47 indicates file closed at 23.21 cf. access
times for R13_46, _45 etc
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 19:09 R13_0
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 19:13 R13_1
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 19:50 R13_10
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 19:54 R13_11
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 19:58 R13_12
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:02 R13_13
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:06 R13_14
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:10 R13_15
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:14 R13_16
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:19 R13_17
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:23 R13_18
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:27 R13_19
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 19:17 R13_2
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:31 R13_20
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:35 R13_21
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:39 R13_22
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:43 R13_23
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:47 R13_24
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:51 R13_25
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:55 R13_26
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 20:59 R13_27
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:03 R13_28
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:08 R13_29
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 19:21 R13_3
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:12 R13_30
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:16 R13_31
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:20 R13_32
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:24 R13_33
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:28 R13_34
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:32 R13_35
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:36 R13_36
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:40 R13_37
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:44 R13_38
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:48 R13_39
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 19:25 R13_4
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:52 R13_40
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 21:56 R13_41
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 22:00 R13_42
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 22:04 R13_43
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 22:08 R13_44
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 22:13 R13_45
-rw-rw-rw-. 1 npg npgstaff 2097152000 Oct 6 22:17 R13_46
-rw-rw-rw-. 1 npg npgstaff 777519104 Oct 6 23:21 R13_47
23.21 R13_47 file closed
23.50 On trying to stop DAQ received nnaida1 timeout error (attachment 9).
DAQ power-cycled, re-started with no data transfer |
|
Wed Oct 5 07:29:29 2016, CG & TD, Wednesday 5 October (inc AIDA+BRIKEN photos) 11x
|
14.30 Adjusted and checked insertion of AIDA snout into BRIKEN (CH2)n moderator.
up-down AIDA snout appears centred (@ (CH2)n centre) and tracks into (CH2)n moderator with no
significant vertical movement
left right AIDA snout appears centred (@ (CH2)n centre) and tracks into (CH2)n moderator with ~3mm
horizontal movement left to right (looking upstream)
15.28 DSSSD bias/leakage current OK - see attachments 1-2
FEE64 temperatures OK - see attachment 3
B2F/F11 temperature +24.8 deg C, d.p. +11.8 deg C, RH 44.4%
16.00 Power off to place ac mains distribution unit between AIDA and BRIKEN
18.34 Power up PCS request for remote data/diagnostics collection |
|
Sun Oct 2 15:03:11 2016, CG, TD, Problems compiling AIDAsort program on aidas1
|
I have been trying to compile the sort program I have been working on in Edinburgh so I can work on it locally
while at RIKEN.
When compiling i receive the error shown in make.log (attached) saying that there are undefined reference in
DataSource.cpp.
Also attached is the make output when compiled in Edinburgh - basically just 'unused variable' warnings.
This is exactly the same DataSource.cpp I am using in Edinburgh. I have carried out 'diff' on the DataSpyLib and
DataXferLib directories which are included
in the Makefile and no differences are present. I have also done this for DataSource directory and again, there
are no differences between what I am running here and in Edinburgh.
The only difference I have been able to observe so far is that in Edinburgh I am using g++ version:
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/4.8.5/lto-wrapper
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-linker-build-id --with-linker-hash-style=gnu
--enable-languages=c,c++,objc,obj-c++,java,fortran,ada,go,lto --enable-plugin --enable-initfini-array
--disable-libgcj --with-isl=/builddir/build/BUILD /gcc-4.8.5-20150702/obj-x86_64-redhat-linux/isl-install
--with-cloog=/builddir/build/BUILD/gcc-4.8.5-20150702/obj-x86_64-redhat-linux/cloog-install
--enable-gnu-indirect-function --with-tune=generic --with-arch_32=x86-64 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.5 20150623 (Red Hat 4.8.5-4) (GCC)
whereas on aidas1:
Using built-in specs.
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix
--enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk
--disable-dssi --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre --enable-libgcj-multifile
--enable-java-maintainer-mode --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --disable-libjava-multilib
--with-ppl --with-cloog --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.4.7 20120313 (Red Hat 4.4.7-17) (GCC)
The libraries used are all found in LD_LIBRARY_PATH.
On aidas1 this is: /MIDAS/Linux/lib64:/homes/npg/root/lib:/homes/npg/root/lib:/MIDAS/Linux/lib64:/homes/npg/root/lib
and in Edinburgh:
/Disk/ds-sopa-group/np/RIKEN/AIDAsort/MIDAS/Linux/lib64:/Disk/ds-sopa-group/np/thehubdata/thehub6/jlab_software/Linux_Scientific6-x86_64-gcc4.4.6/root/5.34.23/lib:
*********UPDATE 03/10/16 *************
The methods are defined in /MIDAS/DataPackage/DataXferLib/V4_TCP/transfer.c
This is in the expected INCLUDES path of: INCLUDES= -I/MIDAS/DataPackage/DataXferLib/V4_TCP -I/MIDAS/DataPackage/DataSpyLib |
|
Sun Oct 2 06:12:54 2016, TD, Sunday 2 October 16x
|
14.15 DSSSD bias/leakage current OK - see attachments 1-2
FEE64 temperatures OK - see attachment 3
B2F/F11 temperature +27.5 deg C, d.p. +10.6 deg C, 34.3 RH %
14.24 R8 starts
ASIC settings 2016Sep02-18.17.01
Master MACB serial #13 (see https://elog.ph.ed.ac.uk/AIDA/367)
Master MACB mode 5
NIM 50MHz -> Phillips 726 NIM-TTL-NIM -> Master MACB - external clock (rear panel)
Note Fuji Diamond Tool Ltd 1020 NIM-TTL level adaptor (NIM, pink front panel) with signal
rise and falltimes quoted as <10ns produced poor quality 25/50MHz clock - the Phillips 726
has significantly faster rise and falltimes and drices a higher amplitude signal to 50R load.
BNC PB-4 trigger output -> Ortec SCA -> LeCroy 429A (500ns width) -> Master MACB - Correlation DAQ
Accepted Trigger
BNC PB-4 frequency c. 26Hz
Statistics - see attachments 4-14
Master Timestamp - see attachment 15
Correlation Status & Control - see attachment 16
Successive correlation scaler events from R8_0
*** MBS information: block: 120 ptr: 7827 data: 0x868068B4 module: 6 information type: 8 information
index: 0 MBS data:68B4 ts: 0x00000019979ADAF8
*** MBS information: block: 120 ptr: 7829 data: 0x86812F00 module: 6 information type: 8 information
index: 1 MBS data:2F00 ts: 0x00000019979ADAF8
*** MBS information: block: 120 ptr: 7831 data: 0x86820000 module: 6 information type: 8 information
index: 2 MBS data:0000 ts: 0x00000019979ADAF8
*** MBS information: block: 126 ptr: 7005 data: 0x8680FDF4 module: 6 information type: 8 information
index: 0 MBS data:FDF4 ts: 0x0000001997D52FF8
*** MBS information: block: 126 ptr: 7007 data: 0x86812F0E module: 6 information type: 8 information
index: 1 MBS data:2F0E ts: 0x0000001997D52FF8
*** MBS information: block: 126 ptr: 7009 data: 0x86820000 module: 6 information type: 8 information
index: 2 MBS data:0000 ts: 0x0000001997D52FF8
0x2F0EFDF4 - 0x2F0068B4 = 0xE9540 = 955712 * 40ns (assume 25MHz) = 38228480ns = 38.2ms
0x1997D52FF8 - 0x19979ADAF8 = 0x3A5500 = 3822848 * 10ns = 38228480ns = 38.2ms
=> correlation clock frequency = 25MHz
Master MACB (serial #13) mode = 5, 50MHz clock to external clock input (rear panel)
clock output 25MHz
reset output reset not observed at DAQ GO
reset observed using Correlation status & control tab -> Force
Correlation Reset
reset request output not tested
correlation DAQ accepted trigger output c.26Hz
Could not persuade MACB serial #13 to operate in mode 0, external clock input (rear panel) disconnected)
Persistent failures to ReSYNC by nnaida11, 12, 17, 18, 19, & 20 (see https://elog.ph.ed.ac.uk/AIDA/357)
following RESET/SETUP, FEE64 power cycle, MACB NIM bin power cycle and re-seating master MACB HDMI cables
Switch back to previous Master MACB serial #11
Master MACB (serial #11) mode = 0
clock output 25MHz
reset output reset not observed at DAQ GO
reset not observed using Correlation status & control tab -> Force
Correlation Reset
reset request output not tested
correlation DAQ accepted trigger output not observed |
|