|
|
|
AIDA
GELINA
BRIKEN
nToF
CRIB
ISOLDE
CIRCE
nTOFCapture
DESPEC
DTAS
EDI_PSA
179Ta
CARME
StellarModelling
DCF
K40
|
| AIDA |
 |
|
|
|
Message ID: 417
Entry time: Thu Nov 3 01:25:30 2016
|
| Author: |
TD, CG |
| Subject: |
Thursday 3 November |
|
|
10.20 DAQ found stalled
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
AIDA SYNC stats - see attachment 4 - stats 10.1M/381/3600 => DAQ stalled at 23.05 + 7.4h => c.06.29
System wide checks OK *except* SYNC pulses & SYNC errors - see attachments 5-8
Conclude DAQ stalling at c. 06.30/18.30 due to start of ORNL LN2 dewar refill cycle
but unrelated to either the BRIKEN-supplied 50MHz TTL clock signal to external clock
input of Master MACB or the ground connection provided by this coaxial cable.
Which implies the problem is either EMI/RFI or ac mains electrical ground.
AIDA powered by two single phase outlets from one of four sections of a single phase ac mains distribution
box with power supplied by two heavy duty (3cm dia) cables from Denkenseiki Noisecut AVR type PMN (AVR4, B3F
104V/51A). AIDA is connected to both NEMA 5-15R-style outlets of section A, LN2 refill to section D, BRIKEN
to sections B, C and D.
Ambient temperature +24.8 deg C, d.p. +4.1 deg C, RH 26.3%
15.00 DSSSD cabling drain wire attachment to AIDA adaptor PCB (FEE64 # 5 & 6) not tightly secured.
CG re-secured drain wire and this has significantly reduced 100kHz period transient noise
as observed at preamplifier outputs.
14.48 Restart DAQ (R110)
Master MACB mode 5, 50MHz TTL clock to external clock input
14.57 DAQ stalled
All system wide checks OK *except* SYNC errors - see attachment 9
ASIC clock timestamp - nnaida19 fails
N.B. SYNC pulses check OK, counters incrementing between successive tests
The latter is a problem we have seen previously but has not stopped DAQ running OK
aidas1> ./analyser /TapeData/Sept2016/R110_2
*** TDR format 3.3.0 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 11861
ADC data format: 82556471 ( 877427.0 Hz)
Other data format: 14217428 ( 151105.7 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 27 ( 0.3 Hz)
RESUME: 27 ( 0.3 Hz)
SYNC100: 861083 ( 9151.8 Hz)
FEE64 disc: 13353468 ( 141923.4 Hz)
MBS info: 2823 ( 30.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 1976 ( 21.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 94.089 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.084 0.000
2 0.076 0.000
3 0.000 0.000
4 0.000 0.000
5 0.109 0.000
6 0.012 0.000
7 0.000 0.000
8 0.000 0.000
9 0.003 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.032 0.000
16 0.013 0.000
17 0.000 0.000
18 0.000 0.000
19 0.125 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.142 0.149
24 0.076 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS Other
HEC Data
0 0 0 0 0 0 0 0 0 0 0
0
1 942953 5806860 0 0 3 3 35855 5770999 0 0
42
2 4036003 4268394 0 0 2 2 35861 4232529 0 0
0
3 3038422 39132 0 0 0 0 35892 3240 0 0
0
4 1192440 42063 0 0 0 0 35889 6174 0 0
68
5 5685966 39310 0 0 5 5 35849 3451 0 0
0
6 3670394 46605 0 0 2 2 35885 7893 2823 0
70
7 2661022 43508 0 0 0 0 35890 7618 0 0
231
8 2165335 41382 0 0 0 0 35890 5492 0 0
359
9 3037520 863964 0 0 1 1 35889 828073 0 0
0
10 2616647 179334 0 0 0 0 35889 143445 0 0
0
11 1083633 39064 0 0 0 0 35889 3175 0 0
0
12 1759629 38368 0 0 0 0 35888 2480 0 0
0
13 2419715 280498 0 0 0 0 35891 244607 0 0
0
14 3189205 1325559 0 0 0 0 35887 1289672 0 0
747
15 4886680 39482 0 0 2 2 35871 3607 0 0
7
16 5087977 38869 0 0 1 1 35886 2981 0 0
0
17 2905638 329411 0 0 0 0 35888 293523 0 0
0
18 3327386 46016 0 0 0 0 35888 10128 0 0
0
19 6770177 40711 0 0 4 4 35842 4861 0 0
7
20 2225755 38524 0 0 0 0 35891 2633 0 0
344
21 2061728 40299 0 0 0 0 35887 4412 0 0
90
22 3052329 512109 0 0 0 0 35890 476219 0 0
7
23 9325689 38673 0 0 3 3 35834 2833 0 0
3
24 5414228 39293 0 0 4 4 35862 3423 0 0
1
25 0 0 0 0 0 0 0 0 0 0
0
26 0 0 0 0 0 0 0 0 0 0
0
27 0 0 0 0 0 0 0 0 0 0
0
28 0 0 0 0 0 0 0 0 0 0
0
29 0 0 0 0 0 0 0 0 0 0
0
30 0 0 0 0 0 0 0 0 0 0
0
31 0 0 0 0 0 0 0 0 0 0
0
32 0 0 0 0 0 0 0 0 0 0
0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0
*** Program elapsed time: 1.500s ( 7907.333 blocks/s, 494.208 Mb/s)
15.48 Restart aidas1
Power cycle FEE64s
Restart AIDA DAQ
Same problem - DAQ stall (no output from merger) at, or shortly after, Start ASIC Readout with
multiple FEE64s with zero AIDA SYNC rate
*All* system wide checks OK
Cannot recover DAQ control of FEE64s with merger restart - power cycle FEE64s again
23.58 Attempt to change master MACB from MACB module #11 mode 5 to MACB module #13 mode 0
Successive fuse failures of USB ac mains relay
System wide ASIC clock timestamp check nnaida 5/6/7/8 13/14/15/16 21/22/23/24 i.e. first half of MACB tree
Re-seat HDMI cabling from master MACB root module to first half of MACB tree - ReSYNC fails - see attachments
10 & 11
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|