 Mon Oct 31 02:26:32 2016, TD, [How To] Start Runs (October 2016) 12x
|
All AIDA operations are controlled from the Dell T410 Workstation aidas1 using web browsers.
AIDA has multiple, independent data acquisition servers connected by an Ethernet network.
Response times to commands, information updates etc are ~10-60s - please be patient!
How To Start Run
1) Workspace 1 - Firefox web browser - Run Control tab
Select 'Update'
Check DAQ stopped (green 'GO' button displayed)
Check state of *all* data acquisition servers is 'stopped'
Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags
See attachment 1
2) Workspace 2 - Firefox web browser - TapeServer tab
Select 'Reload'
Check TapeServer stopped and disk file closed (green GO button displayed)
See attachment 2
3) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload'
Check Merge Program paused - see banner
Merge State = GOing : paused : xfer enabled : output paused
If not paused select 'Toggle Merge Program Pause State'
See attachment 3
4) Workspace 1 - Firefox web browser - ASIC Control tab
Select 'Reload'
Check 'Act on ALL FEE64 modules?' selected
Check 'Act on ALL ASICs' selected
Select 'Select required function' -> 'Check ASIC Control'
After ~60s a pop-up will appear showing the response from each ASIC of each data acquisition server.
Check all ASICs are OK and status = 0x001F0000.
Dismiss pop-up by selecting OK
See attachments 4-5
5) Workspace 1 - Firefox web browser - Run Control tab
Start DAQ - select green GO button
Wait (~60s) until state of all data acquisition servers (nnaida1, nnaida2 ... etc) updates
Check state of *all* data acquisition servers is 'going'
Check histogramming and data transfer #1 is enabled for *all* data acquisition servers as indicated by 'H X' tags
See attachment 6
6) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload' several times and check *all* enabled data links (0-23) are active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are inactive call for expert help and do NOT proceed!
Observe active spinner in MERGER server terminal window
See attachment 7
7) Workspace 2 - Firefox web browser - TapeServer tab
Check 'Volume Name' and 'File Name'
Start TapeServer and open disk file - select green 'GO' button
See attachment 8
8) Workspace 2 - Firefox web browser - MERGE tab
Select 'Reload' several times and check all enabled data links (0-23) are always active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are *always* inactive call for expert help and do NOT proceed!
Select 'Toggle Merge Program Pause State' to enable data merging
Select 'Reload' to observe merge rate (~ 9000 data items/s)
Observe active spinners in MERGE and TapeServer server terminal windows adjacent to web browser
See attachment 9
9) Workspace 1 - Firefox web browser - Run Control tab
Select 'Start ASIC Data Readout'
See attachment 10
10) Workspace 2 - Firefox web browser - MERGE tab
Select Reload several times and check all enabled data links (0-23) are active
bright green = active, olive green = inactive, red = disabled
N.B. If some data links are *never* active call for expert help and do NOT proceed!
Select 'Reload' to observe merge rate (~ 0.5-2M data items/s)
See attachment 11
11) Workspace 2 - Firefox web browser - TapeServer tab
Select 'Reload' and observe number of blocks written to disk increasing
See attachment 12
12) Workspace 1 - Firefox web browser - Control tab
Select 'Save/Restore Settings'
From 'Save/Restore Module Settings' tab
select 'Reload'
select 'Module Names' whose settings should be saved (usually all)
select 'Generate new DB key' and make a note of the DB key generated
the DB key will have the format YYYYMMMDD-HH.MM.SS
The DB key will be used to name a directory to store AIDA settings
and is created uid: npg, gid: npgstaff, perms 755. However, its
contents are written uid: nfsnobody, gid nfsnobody. The default
directory (DB Key) permissions do not work - change the permissions
to 777 - before attempting to save data to this directory (DB Key).
launch a terminal and change the permissions of the DB key directory using the command
chmod 777 /MIDAS/DB/EXPERIMENTS/AIDA/[YYYYMMMDD-HH.MM.SS]
select 'Save Settings' to save AIDA ASIC, LED and waveform settings to
the directory (DB key)
11) Update BRIKEN Elog with the following information
AIDA
run filename (e.g. R99)
[# data blocks]
start date & time
[stop date & time]
ASIC settings DB key
additional comments
your name
Complete run sheet for written logbook |
|
Mon Oct 31 02:00:01 2016, TD, Monday 31 October 11x
|
11.00 DAQ continues OK
DAQ STOP OK
Most recent files R91_568 last accessed 10.59
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System wide checks OK *except* SYNC pulses & errors - see attachments 4-7
14.20 Per FEE64 screenshots p+n junction and n+n ohmic strip waveforms - see attachments 8-11
Preamplifier output noise looks OK for all FEE64s *except* nnaida5 & nnaida6 (which
have significantly higher ADC and disc rates than other FEE64s)
21.10 DAQ found stalled
Most recent file R94_99 last accessed 18.35
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
21.16 DAQ start (R96) |
|
Sun Oct 30 04:37:31 2016, TD, Spectrum layouts created 8x
|
The following spectrum layouts have been created
1 All Rate spectra
2 All HitRate Spectra
3 p+n junction strip 1.8.L spectra
4 n+n ohmic strip 1.8.L spectra
5 p+n junction strip 1.8.H spectra
6 n+n ohmic strip 1.8.H spectra
5 p+n junction strip 1.8.W waveforms
6 n+n ohmic strip 1.8.W waveforms
The layouts can be found in the directory ~/LayOut and should be copied to /tmp
N.B Currently you cannot restore a layout to a spectrum display with > 16 galleries.
If spectrum display uses > 16 galleries, switch spectrum display to 16 galleries
or less before restoring a spectrum layout. |
|
Sun Oct 30 03:26:55 2016, TD, Report - medium - cannot restore layout if currently displaying >16 galleries
|
Cannot restore layout if currently displaying >16 galleries
Obtain the error message 'loading Spectrum.tcl > can't read "dbtagx(31,1)": no such element in array' |
|
Sun Oct 30 01:53:13 2016, TD, Sunday 30 October 9x
|
10.53 DAQ found stalled
Most recent files R89_102 last accessed 07.17
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
All system wide checks OK *except* Clock status, SYNC errors & SYNC pulses - see attachments 4-8 - unusual
messages/reports
One merger error messages - link 18 - see attachment 9
Restart merger
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
Standard soft reset did not work - one or more FEE64s dropped data transfer #1 link at DAQ GO
Retry - one, more FEE64s indicate error state - run control tab unresponsive
Power cycle FEE64s |
|
Sat Oct 29 07:26:59 2016, TD, AIDA ADC & disc data synchronisation check 14x
|
Following analysis of AIDA ADC & disc readout ( see https://elog.ph.ed.ac.uk/AIDA/404 )
changed timestamp resolution from 1us/channel to 20.48us/channel
Offline sort (from disk file)
Attachments 1-3 s1-s32 # ADC data items per FEE64, s100-132 # disc data items per FEE64 (20.48us/channel)
Attachments 4-5 s1000 4 * ( module # - 1 ) + ASIC # versus timestamp (ch 0-95 ADC data, ch 100-195 disc data,
20.48us/channel)
Attachments 6-7 MIDASsort terminal session logs for *offline* sort
Online sort (from TapeServer)
Attachment 8 s1-s32 # ADC data items per FEE64, s100-132 # disc data items per FEE64 (20.48us/channel)
Attachments 9-10 s1000 4 * ( module # - 1 ) + ASIC # versus timestamp (ch 0-95 ADC data, ch 100-195 disc data,
20.48us/channel)
Attachments 11-12 MIDASsort terminal session logs for *online* sort
Attachment 13 MIDASsort FORTRAN sort routine sync.f
Attachment 14 Spectrum titles for the above sort routine
Note
1) data is discontinuous as a function of timestamp because we are sampling buffers
from the TapeServer rather than sorting all data as above
2) a low rate of timewarps is reported which appears to be an artifact of the buffer
sampling - no timewarps reported by analyses of the corresponding data files
|
|
Sat Oct 29 03:31:03 2016, TD, Saturday 29 October 11x
|
11.30 DAQ continues (R87)
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 1-4
11.36 DSSSD bias & leakage currents OK - see attachments 5-8
FEE64 temperatures OK - see attachment 9
16.46 DAQ continues (R87)
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 5-8
18.55 DAQ stalled
Most recent file R87_274 last accessed 18.47
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
19.10 DAQ starts (R89) |
|
Fri Oct 28 06:12:50 2016, TD, Friday 28 October 12x
|
14.12 DAQ stalled as expected
Most recent file R86_245 last accessed 06.26
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
Some merger errors messages - all link 4 - see attachment 4
All system wide checks OK *except SYNC errors & SYNC pulses - see attachments 5-8
Restart merger
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
14.34 DAQ started (R87)
19.50 DAQ continues (R87)
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 9-12 |
|
Thu Oct 27 10:57:49 2016, TD, AIDA data readout
|
Developing real-time AIDA data synchronisation test using MIDASsort
To illustrate AIDA readout, ADC and discriminator data from a pulser event is plotted as a function
of the 100MHz AIDA internal timestamp (1us/channel)
Attachment 1 - nnaida8 number of ADC and discriminator data items
Attachment 2 - nnaida1-24 number of ADC and discriminator data items
Attachment 3 - 4*(FEE64 module # - 1) + ASIC # (ADC channels 0-95, disc channels 100-195) versus timestamp
(1us/channel)
The spectra clearly illustrate
- 500kHz ADC readout cycle
- discriminator readout precedes ADC readout by ~18us (this figure will be shaping time dependent)
- after-pulse due to pulser injection via simple charge terminator (no pole zero correction)
~84us after start of ADC readout (this figure will be pulser decay time and shaping time dependent)
Note events were identified (delimited) by time periods > 2us between successive ADC data items |
|
Thu Oct 27 10:54:14 2016, TD, Thursday 27 October
|
14.20 DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
14.35 DAQ restarted for test purposes
18.55 DAQ stalled as expected
Most recent file R83_125 last accessed 18.30
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
20.48 DAQ starts (R86)
LEC/MEC fast comparator 0xff -> 0x20 |
|
Tue Oct 25 02:06:34 2016, TD, Tuesday 25 October 13x
|
10.10 DAQ found stopped
Most recent file R81_169, last accessed 06.26
R81_169 ADC data synchronised
aidas1> ./analyser /TapeData/Sept2016/R81_169
*** TDR format 3.3.0 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 8369
ADC data format: 67712004 ( 1105598.7 Hz)
Other data format: 570667 ( 9317.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 17 ( 0.3 Hz)
RESUME: 17 ( 0.3 Hz)
SYNC100: 560442 ( 9150.9 Hz)
FEE64 disc: 8352 ( 136.4 Hz)
MBS info: 1839 ( 30.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 81 ( 1.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 61.245 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 0.000
2 0.000 0.000
3 0.000 0.000
4 0.000 0.000
5 0.244 0.254
6 0.181 0.191
7 0.000 0.000
8 0.000 0.000
9 0.000 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.002 0.000
16 0.000 0.000
17 0.001 0.000
18 0.002 0.000
19 0.003 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.085 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS
Other HEC Data
0 0 0 0 0 0 0 0 0 0
0 0
1 229453 23628 0 0 0 0 23359 269 0
0 22
2 2464367 23972 0 0 0 0 23362 610 0
0 0
3 1700233 23656 0 0 0 0 23358 298 0
0 0
4 650519 23359 0 0 0 0 23359 0 0
0 0
5 15435615 25773 0 0 4 4 23267 2498 0
0 0
6 15768889 25301 0 0 3 3 23291 165 1839
0 0
7 963142 23361 0 0 0 0 23361 0 0
0 0
8 773768 23661 0 0 0 0 23361 300 0
0 0
9 2244651 23523 0 0 0 0 23359 164 0
0 0
10 1833324 24095 0 0 0 0 23362 733 0
0 0
11 276357 23657 0 0 0 0 23362 295 0
0 0
12 542691 23359 0 0 0 0 23359 0 0
0 0
13 1695447 23996 0 0 0 0 23363 633 0
0 0
14 2181674 23530 0 0 0 0 23361 169 0
0 0
15 2203091 23362 0 0 1 1 23360 0 0
0 0
16 2179412 23361 0 0 0 0 23361 0 0
0 0
17 2317043 23549 0 0 1 1 23359 188 0
0 0
18 2579367 23969 0 0 1 1 23356 611 0
0 0
19 2589829 23664 0 0 3 3 23358 300 0
0 0
20 1296289 23357 0 0 0 0 23357 0 0
0 0
21 1624570 24024 0 0 0 0 23361 663 0
0 59
22 513131 23525 0 0 0 0 23360 165 0
0 0
23 3691921 23335 0 0 4 4 23327 0 0
0 0
24 1957221 23650 0 0 0 0 23359 291 0
0 0
25 0 0 0 0 0 0 0 0 0
0 0
26 0 0 0 0 0 0 0 0 0
0 0
27 0 0 0 0 0 0 0 0 0
0 0
28 0 0 0 0 0 0 0 0 0
0 0
29 0 0 0 0 0 0 0 0 0
0 0
30 0 0 0 0 0 0 0 0 0
0 0
31 0 0 0 0 0 0 0 0 0
0 0
32 0 0 0 0 0 0 0 0 0
0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0
*** Program elapsed time: 0.957s ( 8744.751 blocks/s, 546.547 Mb/s)
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks OK *except* SYNC errors & SYNC pulses - see attachments 4-7
Merger error messages - see attachment 8
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
All system-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.22 DAQ starts (R82)
16.07 DAQ continues OK
All system-wide checks OK
17.16 DAQ continues OK
All system-wide checks OK
18.27 All system wide checks OK *except* SYNC pulses - see attachment 9
18.34 DAQ stalled
Most recent file R82_125 last accessed 18.30
All system wide checks OK *except* SYNC errors and SYNC pulses - see attachments 10-13
18.47 AIDA -> safe state
FEE64 power OFF
DSSSD bias OFF |
|
Mon Oct 24 02:46:57 2016, TD, Monday 24 October 19x
|
10.47 DAQ found stopped, no merger error messages
Most recent file R78_281, last accessed 06.46
R78_281 ADC data synchronised
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks - see attachments 4-11
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
11.07 DAQ starts (R79)
18.39 DAQ OK
System wide checks OK *except* SYNC errors & SYNC pulses - see attachments 12-15
18.47 DAQ stalls
Most recent file R79_182, last accessed 18.46
System wide checks OK *except* SYNC errors & SYNC pulses - see attachments 16-19
18.56 DAQ starts (R80) |
|
Sun Oct 23 10:22:36 2016, TD, Report - high - merger SEGV
|
Merger crashed during operation - see https://elog.ph.ed.ac.uk/AIDA/399 |
|
Sun Oct 23 02:34:12 2016, TD, Sunday 23 October
|
10.34 DAQ found stopped, no merger error messages
Most recent file R74_296, last accessed 06.31
R70_315 ADC data synchronised
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
Some merger errors - see attachment 4
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.42 DAQ start (R75)
nnaida11 drops data transfer #1 tag at SAQ GO
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.46 DAQ start (R76)
18.18 DAQ found stopped
Most recent file R76_29 last accessed 11.52
Merger crashed - see attachment 5
aidas1> ./analyser /TapeData//Sept2016/R76_29
*** GREAT format 3.2.2 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 24384
ADC data format: 197754460 ( 1539809.2 Hz)
Other data format: 1194596 ( 9301.7 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 74 ( 0.6 Hz)
RESUME: 74 ( 0.6 Hz)
SYNC100: 1172671 ( 9131.0 Hz)
FEE64 disc: 18000 ( 140.2 Hz)
MBS info: 3777 ( 29.4 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 190 ( 1.5 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 128.428 s
FEE elapsed dead time(s)
1 0.000
2 0.007
3 0.000
4 0.000
5 0.674
6 2.583
7 0.000
8 0.000
9 0.000
10 0.000
11 0.000
12 0.000
13 0.000
14 0.000
15 0.276
16 0.124
17 0.435
18 0.050
19 0.622
20 0.009
21 0.000
22 0.000
23 0.352
24 0.000
25 0.000
26 0.000
27 0.000
28 0.000
29 0.000
30 0.000
31 0.000
32 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS
Other HEC Data
0 0 0 0 0 0 0 0 0 0
0 0
1 566974 49807 0 0 0 0 48983 824 0
0 62
2 5617753 50240 0 0 2 2 48984 1252 0
0 0
3 4298949 49615 0 0 0 0 48983 632 0
0 0
4 1980475 48985 0 0 0 0 48985 0 0
0 0
5 31381665 53750 0 0 12 12 48726 5000 0
0 0
6 35397222 52131 0 0 12 12 48004 326 3777
0 0
7 3679182 48987 0 0 0 0 48987 0 0
0 0
8 3893833 49613 0 0 0 0 48983 630 0
0 0
9 4909194 49313 0 0 0 0 48987 326 0
0 0
10 4099868 50957 0 0 0 0 48984 1973 0
0 0
11 1590517 49623 0 0 0 0 48987 636 0
0 0
12 2103665 48985 0 0 0 0 48985 0 0
0 0
13 3909540 50385 0 0 0 0 48990 1395 0
0 0
14 5011678 49344 0 0 0 0 48988 356 0
0 0
15 8074527 48904 0 0 10 10 48883 1 0
0 2
16 8822855 48949 0 0 5 5 48939 0 0
0 0
17 11581998 49241 0 0 10 10 48820 401 0
0 0
18 17158740 50227 0 0 1 1 48968 1257 0
0 0
19 14684270 49399 0 0 11 11 48747 630 0
0 0
20 6793305 47946 0 0 1 1 47944 0 0
0 0
21 4052765 50377 0 0 0 0 48986 1391 0
0 126
22 1773964 49339 0 0 0 0 48988 351 0
0 0
23 10790840 48872 0 0 10 10 48852 0 0
0 0
24 5580681 49607 0 0 0 0 48988 619 0
0 0
25 0 0 0 0 0 0 0 0 0
0 0
26 0 0 0 0 0 0 0 0 0
0 0
27 0 0 0 0 0 0 0 0 0
0 0
28 0 0 0 0 0 0 0 0 0
0 0
29 0 0 0 0 0 0 0 0 0
0 0
30 0 0 0 0 0 0 0 0 0
0 0
31 0 0 0 0 0 0 0 0 0
0 0
32 0 0 0 0 0 0 0 0 0
0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undef-d Samples
0 0 0 0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Program elapsed time: 3.586s ( 6799.896 blocks/s, 424.993 Mb/s)
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
18.50 DAQ starts (R77)
19.17 DAQ stalled due to disk housekeeping
Restart merger OK
DAQ STOP fails - nnaida1 timeout
Power cycle FEE64s
19.47 DAQ starts (R78) |
|
Sat Oct 22 02:32:37 2016, TD, Saturday 22 October 15x
|
10.32 DAQ found stopped, no merger error messages
Most recent file R70_315, last accessed 06.46
R70_315 ADC data synchronised
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks - SYNC errors & SYNC pulses fail - other checks OK - see attachments 4-7
AIDA SYNC stats - see attachment 8
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
10.42 DAQ starts (R71)
18.25 DAQ stop OK
System wide test all OK *except* SYNC errors - see attachments 8-11
New SYNC error baseline
18.54 Repeat system wide checks whilst DAQ is still halted - SYNC errors *and* SYNC pulse now fail - see
attachments 12-15
This may suggest that the clock signal itself is affected by LN2 fill cycle
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
18.59 DAQ starts (R72)
nnaida15 drops data transfer #1 tag at DAQ GO
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
19.07 DAQ starts (R73)
19.17 DAQ stalled
Restart merger OK
DAQ STOP OK
Syten wide checks - SYNC error and SYNC pulse fail - dissimilar FEE64 counts in SYNC pulse
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
19.37 DAQ starts (R74)
|
|
Fri Oct 21 01:41:54 2016, TD, Friday 21 October 14x
|
09.38 DAQ (R66) continues OK
Detector bias and leakage currents OK - see attachments 1&2
FEE64 temperatures OK - see attachment 3
System wide checks SYNC count & SYNC errors fail - see attachments 4-7 - other checks OK
R66_367 ADC data synchronised
AIDA SYNC stats - attachment 8
Some merger errors - see attachment 9
13.53 DAQ found stalled
Most recent files R66_396 last accessed 10.55
System wide checks SYNC count & SYNC errors fail - see attachments 10-13 - other checks OK
AIDA SYNC stats - attachment 14
No further merger error messages
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
14.02 DAQ starts (R67)
18.27 DAQ found stalled
Most recent file R67_121 last accessed 18.26
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
18.39 DAQ start (R68)
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
18.44 DAQ stalled
Most recent file R68_1 last accessed 18.44
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
18.53 DAQ starts (R70) |
|
Thu Oct 20 20:00:14 2016, CG, Returned module tests @ DL 7x
|
Original system configuration before my arrival: nnaida11 - 2b:22:6e
nnaida12 - 2b:33:15
nnaida13 - 2b:09:ce
All on firmware version 0x1900c1f.
With 8us shaping time and pulser amplitude @ 5V, FWHM [ch] were:
Ch | nnaida11 | nnaida12 | nnaida13 | nnaida14
1.4.L | 38 | 33 | 38 | 34
1.6.L | 24 | 25 | 28 | 23
1.8.L | 20 | 17 | 17 | 16
1.10.L | 21 | 17 | 17 | 16
Slightly higher than expected for a few channels, but FWHM fall within ~16-40ch for all channels in all FEEs.
All system checks passed.
nnaida13/14 kept constant throughout as control.
(1) Swapped nnaida11/12: nnaida11 -> 2b:33:0c nnaida12 -> 2a:d0:1a
Initially failed SYNC error test - consequence of old firmware version.
Updated firmware from 0x10300c12 to 0x1900c1f.
All system tests passed.
FWHM on 8us shaping time and same pulser settings.
Ch | nnaida11 | nnaida12 | nnaida13 | nnaida14
1.4.L | 37 | 40 | 42 | 35
1.6.L | 31 | 28 | 29 | 23
1.8.L | 22 | 18 | 15 | 15
1.10.L | 23 | 20 | 16 | 15
(2) Swapped nnaida11/12: nnaida11 -> 2a:b8:90 nnaida12 -> 2a:b6:45
Updated firmware from 0x10300c12 to 0x1900c1f.
All system checks passed.
No ASIC temp in nnaida11.
FWHM on 8us shaping time and same pulser settings.
Ch | nnaida11 | nnaida12 | nnaida13 | nnaida14
1.4.L | 40 | 32 | 40 | 34
1.6.L | 29 | 25 | 29 | 23
1.8.L | 21 | 18 | 17 | 16
1.10.L | 21 | 18 | 17 | 16
(3) Swapped nnaida11/12: nnaida11 -> 2a:ed:9a nnaida12 -> 2b:11:c5
Updated firmware from 0x10300c12 to 0x1900c1f.
All system checks passed.
No ASIC temp in nnaida12.
FWHM on 8us shaping time and same pulser settings.
Ch | nnaida11 | nnaida12 | nnaida13 | nnaida14
1.4.L | 44 | 36 | 40 | 35
1.6.L | 30 | 27 | 29 | 24
1.8.L | 22 | 18 | 16 | 17
1.10.L | 23 | 18 | 17 | 16
The above were all the repaired modules and seem to be operating normally.
Following are the 2 modules for which no fault report was provided. I only had time to complete testing on one, the other remains unknown.
(4) Swapped nnaida11/12: nnaida11 -> 2a:ed:8f nnaida12 -> 2b:09:da
Updated firmware from 0x10300c12 to 0x1900c1f.
nnaida11 failed clock status tests. Tried 'Perform resync' from Master Timestamp window with no effect.
No SYNC errors reported. All other system tests passed.
nnaida12 missing lots of channels but all system checks passed.
FWHM on 8us shaping time and same pulser settings.
Ch | nnaida11 | nnaida12 | nnaida13 | nnaida14
1.4.L | 46 | 46 | 41 | 36
1.6.L | 31 | 26 | 28 | 24
1.8.L | 22 | 19 | 17 | 16
1.10.L | 22 | 20 | 17 | 16
**
Overall, the first three modules which have been repaired seem to be operating normally. The fourth seems not be and the last module remains unknown.
Ch 1.4.L displays greater than expected FWHM throughout in all nnaida11/12, but FWHM were largely constant (to within a few channels) throughout and all FWHM fell in the range ~15-40ch apart from 1.1.L in *all* nnaida. |
|
Thu Oct 20 01:30:10 2016, TD, Thursday 20 October 12x
|
09.30 DAQ found stopped, no merger error messages
Most recent file R63_291, last accessed 06.46
R63_291 ADC data synchronised
aidas1> ./analyser2 /TapeData/Sept2016/R63_291
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 29203
ADC data format: 237170622 ( 2006571.1 Hz)
Other data format: 1096655 ( 9278.2 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 113 ( 1.0 Hz)
RESUME: 113 ( 1.0 Hz)
SYNC100: 1077835 ( 9119.0 Hz)
FEE64 disc: 15061 ( 127.4 Hz)
MBS info: 3533 ( 29.9 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 175 ( 1.5 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 118.197 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.000 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 2.763 s
FEE module #: 6 elapsed dead time 0.275 s
FEE module #: 7 elapsed dead time 0.004 s
FEE module #: 8 elapsed dead time 0.003 s
FEE module #: 9 elapsed dead time 0.001 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.013 s
FEE module #: 15 elapsed dead time 0.028 s
FEE module #: 16 elapsed dead time 0.275 s
FEE module #: 17 elapsed dead time 6.010 s
FEE module #: 18 elapsed dead time 0.382 s
FEE module #: 19 elapsed dead time 0.555 s
FEE module #: 20 elapsed dead time 0.378 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.035 s
FEE module #: 23 elapsed dead time 0.168 s
FEE module #: 24 elapsed dead time 0.043 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 28.188s ( 1036.027 blocks/s, 64.752 Mb/s)
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System-wide checks - SYNC errors & SYNC pulses fail - other checks OK - see attachments 4-7
AIDA SYNC stats - see attachment 8
Restart merger OK
DAQ STOP OK
DAQ RESET & SETUP OK
ReSYNC OK
System-wide checks OK
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
09.43 DAQ starts (R64)
18.25 DAQ stop OK
Stop before 'scheduled' 18.30 DAQ stall
System wide checks - SYNC count & SYNC errors fail - other checks OK - see attachments 8-11
18.49 DAQ start (R65)
N.B. normal DAQ start - no DAQ RESET, SETUP, restart of merger & tapeserver etc
18.53 SYNC error counter baseline
19.21 DAQ found stalled
Most recent files R65_12 last access 19.17
Soft reset failed - nnaida4 connect timeouts - power cycle FEE64s
20.02 DAQQ start (R66) |
|
Wed Oct 19 01:14:10 2016, TD, Wednesday 19 October 8x
|
09.13 DAQ found stopped, no merger error messages
Most recent file R55_10, last accessed 19.17
Restart merger OK
DAQ STOP OK
aidas1> ./analyser2 /TapeData/Sept2016/R55_10
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 23037
ADC data format: 187091367 ( 2006959.5 Hz)
Other data format: 867516 ( 9306.0 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 62 ( 0.7 Hz)
RESUME: 62 ( 0.7 Hz)
SYNC100: 851796 ( 9137.4 Hz)
FEE64 disc: 12836 ( 137.7 Hz)
MBS info: 2760 ( 29.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 124 ( 1.3 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 93.221 s
FEE module #: 1 elapsed dead time 0.000 s
FEE module #: 2 elapsed dead time 0.000 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.922 s
FEE module #: 6 elapsed dead time 1.113 s
FEE module #: 7 elapsed dead time 0.017 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.001 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.000 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.005 s
FEE module #: 15 elapsed dead time 0.016 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.446 s
FEE module #: 18 elapsed dead time 0.435 s
FEE module #: 19 elapsed dead time 0.925 s
FEE module #: 20 elapsed dead time 0.015 s
FEE module #: 21 elapsed dead time 0.000 s
FEE module #: 22 elapsed dead time 0.000 s
FEE module #: 23 elapsed dead time 0.221 s
FEE module #: 24 elapsed dead time 0.000 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 22.266s ( 1034.644 blocks/s, 64.665 Mb/s)
R55_10 ADC data synchronised
DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
SYNC statistics, System Wide Checks SYNC error & SYNC pulses fail - see attachments 4-8
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
#1 DAQ start (R58)
nnaida7 data transfer #1 tag (X) drops at DAQ GO
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
#2 DAQ start (R59)
nnaida15 data transfer #1 tag (X) drops at DAQ GO
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
09.35 DAQ start (R60)
16.28 DAQ stop OK
Final file R60_169
16.30 DAQ start (R61)
BNC PB-4 pulser OFF
Background run - files R61_0 - R61_31
+ 60Co (ID# 0115) - files R61_33 - R61_49
60Co source placed on top of AIDA snout ~ above DSSSD #3
DAQ stalled at 18.28 (during R61_49) - as expected! See https://elog.ph.ed.ac.uk/AIDA/393
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1
20.05 DAQ start (R63)
BNC PB-4 pulser ON, 60Co source removed |
|
Tue Oct 18 06:56:52 2016, TD, DAQ status
|
From the installation of the RG58 cable for the 50MHz TTL clock from BRIKEN DAQ
to AIDA Master MACB external clock input we observe DAQ stalls at c. 06.30 and
18.30 daily. This coincides with the nominal start times of the ORNL Clover detector
LN2 fill cycles.
run start stop files recovery
AIDA Master MACB internal clock
R1 28.9.16 15.02 11.20 R1_304 OK
RG174 cable from BRIKEN DAQ 50MHz clock (TTL) to AIDA Master MACB external clock input
9.10.16 16.55 17.30 no data transfer, TS_SYNC phase = 0, OK
9.10.16 17.40 19.15 est no data transfer, TS_SYNC phase = 0, ReSYNC
9.10.16 22.20 01.40 no data transfer, TS_SYNC phase = 1, ?
10.10.16 14.20 16.50 no data transfer, TS_SYNC phase = 2, ?
10.10.16 18.35 18.40/19.40 no data transfer, TS_SYNC phase = 3, some FEE64s stopped later than others, ReSYNC
10.10.16 20.36 06.35 no data transfer, TS_SYNC phase = 0, power cycle
RG174 cable from BRIKEN DAQ 50MHz clock (TTL) to AIDA Master MACB external clock input
replaced by RG58 cable
run start stop files recovery
R17 11.10.16 15.46 08.22 R17_440 soft reset
R21 12.10.16 14.05 15.30 power dip - power cycle
R22 12.10.16 16.20 19.05 OK
R24 13.10.16 OK
R25 13.10.16 OK
R26 13.10.16 OK
R27 13.10.16 OK
R28 13.10.16 OK
R29 13.10.16 OK
R32 13.10.16 21.20 22.34 R32_26 soft reset
R33 14.10.16 00.01 00.25 ? power cycle
R40 14.10.16 01.49 06.26 R40_91 soft reset
R41 14.10.16 OK
R42 15.10.16 22.45 07.18 R42_164 System-wide tests - SYNC errors fail *but* SYNC pulses OK - ReSYNC only
R43 15.10.16 11.39 12.43 disk housekeeping caused merger/tapeserver problem
R46 15.10.16 12.43 18.26 R46_109 soft reset
R47 16.10.16 11.12 18.26 R47_120 power cycle
R49 16.10.16 21.05 06.26 R49_226 soft reset
R51 17.10.16 12.21 18.30 R51_145 soft reset
R52 17.10.16 20.41 06.30 R52_211 soft reset
R53 18.10.16 10.02 18.30 R53_193 soft reset
R55 18.10.16 18.52 19.17 R55_10 soft reset
R60 19.10.16 09.35 16.28 R60_169 OK
R61 19.10.16 16.30 18.26 R61_69 power cycle
R63 19.10.16 20.05 06.46 R63_291 soft reset
R64 20.10.16 09.45 18.25 R64_240 OK - System-wide tests - SYNC errors fail & SYNC pulses fail - other checks OK
R65 20.10.16 18.49 19.17 R65_12 power cycle
R66 20.10.16 20.02 10.55 R66_396 soft reset auto LN2 fill may have failed, manually selected c. 11.00
R67 21.10.16 14.02 18.26 R67_121 soft reset
R68 21.10.16 18.40 18.44 R68_1 soft reset
R70 21.10.16 18.53 06.46 R70_315 soft reset
R71 22.10.16 10.42 18.25 OK
R73 22.10.16 19.07 19.17 R73_4 soft reset
R74 22.10.16 19.38 06.31 R74_296 soft reset
R76 23.10.16 10.46 11.52 R76_29 merger crashed, soft reset
R77 23.10.16 18.50 19.17 disk housekeeping caused merger/tapeserver problem
R78 23.10.16 19.47 06.46 R78_291 soft reset
R79 24.10.16 11.07 18.46 R79_182 soft reset
R80 24.10.16 18.56 19.17 R80_5 DAQ stall, soft reset
R81 24.10.16 19.29 06.26 R81_169 soft reset
R82 25.10.16 10.22 18.30 R82_125 -> safe state
R83 27.10.16 14.35 18.30 R83_125 soft reset
R86 27.10.16 20.48 06.26 R86_245 soft reset
R87 28.10.16 14.34 18.47 R87_274 soft reset
R89 29.10.16 19.10 07.17 R89_102 power cycle
soft reset => no FEE64 power cycle
Restart merger
DAQ STOP
System wide checks
DAQ RESET & SETUP
System-wide checks
Disable H & data transfer #1
Restart merger & tapeserver
Enable H & data transfer #1 |