 Tue Nov 1 23:55:33 2016, TD, CG, Wednesday 2 November 12x
|
08.56 DAQ continues OK
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
System wide checks OK *except* SYNC pulses - see attachments 4-6
Most recent file R101_249 ADC & disc data synchronised - see attachments 7-8
spectrum s1000 - per ASIC data items as function of timestamp (20us/channel)
spectra 1-32 and 100-132 per FEE64 data items as function of timestamp (20us/channel)
09.03 DAQ stopped
17.10 DAQ started (R103) no storage mode
Master MACB module #1 mode 0, 50MHz TTL clock signal cable disconnected from MACB rear panel.
18.55 Checked DAQ status and it has stopped. No file write times so can't easily check exactly when (see SYNC
stat page to calculate).
All system checks passed except SYNC errors (see attachments 9-12)
Upon attempting to Stop the DAQ, could not contact nnaida11.
AIDA SYNC statistics c. 1.76M SYNCs per FEE64. SYNC rate = 381Hz => elapsed time c. 1.3h => DAQ stalled c.
18.30
23.05 DAQ started (R104) no storage mode
Master MACB module #1 mode 0, 50MHz TTL clock signal cable disconnected from MACB rear panel. |
|
Thu Nov 3 01:25:30 2016, TD, CG, Thursday 3 November 11x
|
10.20 DAQ found stalled
DSSSD bias & leakage current OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
AIDA SYNC stats - see attachment 4 - stats 10.1M/381/3600 => DAQ stalled at 23.05 + 7.4h => c.06.29
System wide checks OK *except* SYNC pulses & SYNC errors - see attachments 5-8
Conclude DAQ stalling at c. 06.30/18.30 due to start of ORNL LN2 dewar refill cycle
but unrelated to either the BRIKEN-supplied 50MHz TTL clock signal to external clock
input of Master MACB or the ground connection provided by this coaxial cable.
Which implies the problem is either EMI/RFI or ac mains electrical ground.
AIDA powered by two single phase outlets from one of four sections of a single phase ac mains distribution
box with power supplied by two heavy duty (3cm dia) cables from Denkenseiki Noisecut AVR type PMN (AVR4, B3F
104V/51A). AIDA is connected to both NEMA 5-15R-style outlets of section A, LN2 refill to section D, BRIKEN
to sections B, C and D.
Ambient temperature +24.8 deg C, d.p. +4.1 deg C, RH 26.3%
15.00 DSSSD cabling drain wire attachment to AIDA adaptor PCB (FEE64 # 5 & 6) not tightly secured.
CG re-secured drain wire and this has significantly reduced 100kHz period transient noise
as observed at preamplifier outputs.
14.48 Restart DAQ (R110)
Master MACB mode 5, 50MHz TTL clock to external clock input
14.57 DAQ stalled
All system wide checks OK *except* SYNC errors - see attachment 9
ASIC clock timestamp - nnaida19 fails
N.B. SYNC pulses check OK, counters incrementing between successive tests
The latter is a problem we have seen previously but has not stopped DAQ running OK
aidas1> ./analyser /TapeData/Sept2016/R110_2
*** TDR format 3.3.0 analyser - TD - October 2016
*** ERROR: READ I/O error: 5002
blocks: 11861
ADC data format: 82556471 ( 877427.0 Hz)
Other data format: 14217428 ( 151105.7 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 27 ( 0.3 Hz)
RESUME: 27 ( 0.3 Hz)
SYNC100: 861083 ( 9151.8 Hz)
FEE64 disc: 13353468 ( 141923.4 Hz)
MBS info: 2823 ( 30.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 1976 ( 21.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 94.089 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.084 0.000
2 0.076 0.000
3 0.000 0.000
4 0.000 0.000
5 0.109 0.000
6 0.012 0.000
7 0.000 0.000
8 0.000 0.000
9 0.003 0.000
10 0.000 0.000
11 0.000 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.032 0.000
16 0.013 0.000
17 0.000 0.000
18 0.000 0.000
19 0.125 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.142 0.149
24 0.076 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 Disc MBS Other
HEC Data
0 0 0 0 0 0 0 0 0 0 0
0
1 942953 5806860 0 0 3 3 35855 5770999 0 0
42
2 4036003 4268394 0 0 2 2 35861 4232529 0 0
0
3 3038422 39132 0 0 0 0 35892 3240 0 0
0
4 1192440 42063 0 0 0 0 35889 6174 0 0
68
5 5685966 39310 0 0 5 5 35849 3451 0 0
0
6 3670394 46605 0 0 2 2 35885 7893 2823 0
70
7 2661022 43508 0 0 0 0 35890 7618 0 0
231
8 2165335 41382 0 0 0 0 35890 5492 0 0
359
9 3037520 863964 0 0 1 1 35889 828073 0 0
0
10 2616647 179334 0 0 0 0 35889 143445 0 0
0
11 1083633 39064 0 0 0 0 35889 3175 0 0
0
12 1759629 38368 0 0 0 0 35888 2480 0 0
0
13 2419715 280498 0 0 0 0 35891 244607 0 0
0
14 3189205 1325559 0 0 0 0 35887 1289672 0 0
747
15 4886680 39482 0 0 2 2 35871 3607 0 0
7
16 5087977 38869 0 0 1 1 35886 2981 0 0
0
17 2905638 329411 0 0 0 0 35888 293523 0 0
0
18 3327386 46016 0 0 0 0 35888 10128 0 0
0
19 6770177 40711 0 0 4 4 35842 4861 0 0
7
20 2225755 38524 0 0 0 0 35891 2633 0 0
344
21 2061728 40299 0 0 0 0 35887 4412 0 0
90
22 3052329 512109 0 0 0 0 35890 476219 0 0
7
23 9325689 38673 0 0 3 3 35834 2833 0 0
3
24 5414228 39293 0 0 4 4 35862 3423 0 0
1
25 0 0 0 0 0 0 0 0 0 0
0
26 0 0 0 0 0 0 0 0 0 0
0
27 0 0 0 0 0 0 0 0 0 0
0
28 0 0 0 0 0 0 0 0 0 0
0
29 0 0 0 0 0 0 0 0 0 0
0
30 0 0 0 0 0 0 0 0 0 0
0
31 0 0 0 0 0 0 0 0 0 0
0
32 0 0 0 0 0 0 0 0 0 0
0
*** Timewarps
FEE ADC Pause Resume SYNC100 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0
*** Program elapsed time: 1.500s ( 7907.333 blocks/s, 494.208 Mb/s)
15.48 Restart aidas1
Power cycle FEE64s
Restart AIDA DAQ
Same problem - DAQ stall (no output from merger) at, or shortly after, Start ASIC Readout with
multiple FEE64s with zero AIDA SYNC rate
*All* system wide checks OK
Cannot recover DAQ control of FEE64s with merger restart - power cycle FEE64s again
23.58 Attempt to change master MACB from MACB module #11 mode 5 to MACB module #13 mode 0
Successive fuse failures of USB ac mains relay
System wide ASIC clock timestamp check nnaida 5/6/7/8 13/14/15/16 21/22/23/24 i.e. first half of MACB tree
Re-seat HDMI cabling from master MACB root module to first half of MACB tree - ReSYNC fails - see attachments
10 & 11
|
|
Sat Nov 19 02:46:09 2016, TD, CG, Saturday 19 November 7x
|
11.45 DSSSD bias & leakage currents OK - see attachments 1 & 2
FEE64 temperatures OK - see attachment 3
Updated copy of GoogleDocs leakage current spreadsheet - attachment 4
20:17 DSSD bias and leakage currents ok (attachments 4+5).
FEE temps ok (attachment 6) |
|
Wed Nov 23 01:47:28 2016, TD, CG, Wednesday 23 November 6x
|
10.45 DSSSD bias & leakage currents - see attachments 1 & 2
note - significant increase in leakage current (~0.8uA) since previous readings
may be temperature transient or charging effect due to delta electrons due to BigRIPS tuning to F11
15cm Pb installed upstream of AIDA+BRIKEN some days ago
FEE64 temperatures OK - see attachment 3
15.35 DSSD bias and leakage currents ok (attachments 4+5).
Marginal increase over this morning.
FEE temps ok (attachment 6).
|
|
Thu Nov 24 01:07:14 2016, TD, CG, Thursday 24 November 29x
|
10.05 DSSSD bias & leakage currents OK - see attachments 1 & 2
note - the significant increase observed yesterday appears to be plateau'ing
FEE64 temperatures OK - see attachment 3
10.19 System-wide tests - see attachments 4-17
Clock status fails for varying combinations of nnaida5, 7, 8, 15 & 16
ADC calibration fails for nnaida23
All other tests OK
10.27 LOCAL controls for nnaida5, 6, 7, 8, 15 & 16 - see attachments 18-23
10.37 Start DAQ (Nov2016/R1)
Analysis of R1_0 demonstrates ADC & disc data synchronised - see attachments 24 & 25
Analysis of R1_0 - see attachment 26
20.10 Successfully restarted the DAQ after fill cycle stop.
Bias and leakage currents ok (attachments 27+28).
Temps ok (attachment 29). |
|
Sun Jun 16 10:06:21 2019, TD, CG, FEE64 nnaida14 noise issues 16x
|
Attachments 1, 3, 5 & 7 *.L spectra log scale
Attachments 2, 4, 6 & 8 *.L spectra linear scale (expanded around pulser peak)
Attachments 9-12 *.W spectra 20us FSR
Observe
2.13.L low amplitude
2.14.L & 3.3.L large number of counts and 'sinusoidal' continuum
... but ... the corresponding waveform spectra look OK
Zero'd all histograms to confirm observations - confirmed - see attachment 16
Pulser peak width spectrum 3.8.L 71 ch FWHM |
|
Fri Nov 17 09:19:41 2023, TD, CC, VP, AIDA test 17/11/2023 46x
|
We restarted AIDA (a10277ttachment 6) this morning and checked rates (attachments 3/4) and waveforms (attachments 1/2). Temperatures are shown in attachment 5.
It seems consistent with what was observed yesterday.
AIDA stopped and platform moved.
The power cable from USB main power relay was connected to the wall and not the DESPEC 19" racks as expected. It has been corrected : it is now connected to the DESPEC 19" rack, has it should be.
AIDA was restarted after that. bPlast was disconnected from 2 booster cards.
We checked the rates (attachments 7,10 and 11 [linear scale]). They are quite similar, although aida01 rate has increased (maybe a cable issue), and rate from aida07 significantly decreased (it was really low yesterday as well).
The waveforms (attachments 8 & 9) are similar as the ones obtained earlier.
bPlast has been reconnected to all booster cards. Rate increased a little (attachment 12) but broadly similar to that observed first thing this morning.
Then 2 booster cards were grounded, and the rates went up (attachments 13-16)
Afterwards, one power cable was connected to the booster card on the short side, increasing the noise, as observed yesterday (attachments 17-20)
The Rohde Schwarz PSU output was referenced to PSU ground ( 3mm cable to ground socket front panel ). It did not reduced much the noise (attachment 21)
Finally, the power supply was switched on, increasing the noise in AIDA (attachment 22).
After the meeting, we have chosen to reconnect bPlast to the booster : now all LIMOs are connected to the booster cards, the cards are powered and the SiPM are biased. The alarm for the short side has triggered (not full bias)
The Rohde & Scwarz PSU outputs is till referenced to its chassis ground.
See attachments 23-26 for the rates and waveforms in AIDA
14.57 JG/JB added additional black cloth to reduce nPlast SiPM dark current : now it is around 16uA. Updated noise observations
per FEE64 1.8.W spectra 20us FSR - attachments 27-28
per FEE64 1Rate spectra - attachment 29
ADC data item stats - attachment 30
aida02 asics #2-4 remain noisy c. kHz/channel
Most other p+n FEE64s are OK - the usual 'hot' channels but most other channel rates low
15.07 JG removed ground reference for Rohde & Schwarz PSU +4V output to bPlast booster cards
ADC data items atst - attachment 31
Most FEE64s have significantly higher rates so the ground reference is doing 'something'
15.12 JG re-ground ref'd Rohde & Schwarz PSU +4V output to bPlast booster cards
ADC data item stats - attachment 32
Rates for most FEE64s are reduced - some significantly - PSU ground ref is doing something reproducible
15.25 Christoph Scheidenberger reports FRS magnets ON
16.01 JG/JB TAMEX & NIM bin electrically isolated from AIDA stand/DESPEC platform. Screens of flat cables from bPlast to booster cards grounded to snout.
ADC data item stats - attachment 33
per FEE64 1.8.W spectra 20us FSR - attachments 36-37
per FEE64 Rate spectra - attachments 34-35
16.20 TAMEX & NIM bin electrically isolated from AIDA stand/DESPEC platform. Screens of flat cables from bPlast to booster cards now connected together but electrically floating
ADC data item stats - attachment 40
per FEE64 1.8.W spectra 20us FSR - attachments 38-39
per FEE64 Rate spectra - attachment 41
Noted improvement in most FEE64 rates - aida04 improved significantly
16.31 TAMEX & NIM bin electrically isolated from AIDA stand/DESPEC platform. Screens of flat cables from bPlast to booster cards now connected together and grounded to NIM bin chassis.
ADC data item stats - attachment 42
per FEE64 Rate spectra - attachment 43-44
per FEE64 1.8.W spectra 20us FSR - attachments 44-46
Continued, imcremental improvement in FEE64 rates |
|
Fri May 19 02:44:14 2017, TD, CB, OH, Friday 19th May
|
10:44 System wide checks all ok
Leakages and biases all ok (see attachment 1)
Ambient Temperature 24.4C - Humidity 38.1% - Dew Point +9.1
FEE temperatures ok (see attachment 2)
DAQ, Merger and Tape server all still running ok
Good events statistics (see attachment 3)
Orientation of RS-232 cards was wrong. They have now been corrected will enable logging then do a halt and
go on the daq.
Power supply also moved onto AIDA circuit.
11:00 Current file R15_333.
11:04 halt and go performed
24 Log files produced on Rasp Pi console monitor. |
|
Sun May 28 02:16:04 2017, TD, CB, Sunday 28 May 16x
|
10.15 DAQ continues OK
Current file May2017/R31_239
Detector biases; leakage currents OK - see attachments 1 & 2
FEE64 Temperatures OK - see attachment 3
good events; statistics OK - see attachment 4
No warnings/errors reported by merger since DAQ GO - see attachment 5
*all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)
10.30 E17/F11 ambient temperature +23.8 deg C, RH 40.9%, d.p. +9.6 deg C
Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 95% - refilled)
10.35 Layouts 1, 3, 4, 7 and 8 - see attachments 6-13
Note - as expected, zero HEC events, layouts 5 & 6 not shown
10.58 DAQ stop - transient timeout nnaida1 - followed by successful stop
File May2017/R31_248
Delete R31
11.12 Disable data transfer
DAQ GO - transient timeout nnaida1 - followed by successful start
nnaida1 system console - see attachment 14
12:00 Moved plastic veto scintillator downstream in an attempt to reduce noise
Rates in noisy FEEs reduced by 20-30%
14:30 DAQ continues OK
Found that pulser induces counts in some HEC (nnaida20, 14 in particular). See attachment 15. Pulser off
removes these events completely.
These events could be mistaken for implantation events during data acquisition.
Increasing the fast comparator threshold HEC from 2 -> 3 suppresses the events. Increasing the threshold
to 4 does not appear to have further effects.
Even with the increased threshold some HEC (e.g. 1.0H) still see the pulser. See attachment 16 (fast
comparator threshold HEC was 0x3).
Checked data is synchronised in file R32_33
15:29 Set the threshold back to 2. We will perform this cut off-line in the analysis. |
|
Tue May 30 00:58:10 2017, TD, CB, Tuesday 30 May - day shift 27x
|
08.58 DAQ continues OK
File RIBF127R1/R1_99
BigRIPS run 1009
BRIKEN run 9
BNC PB-4
Amplitude 90,000
x5 attenuator IN
t_r 100ns tau_d 50us
- polarity
frequency 25Hz range
delay MIN, tail pulse, INT ref
ASIC settings 2017Mar27-16.12.39
08.59 *online* analysis - ADC & disc data synchronised - see attachment 1
analysis of R1_99 OK - see attachment 2 - deadtime <0.5%
09.11 BigRIPS & BRIKEN DAQ stop
09.13 AIDA RIBF127R1/R1_103
BRIKEN 10
BigRIPS 1010
Fixed Degrader in place 13m Al (8mm + 3mm + 2mm Plates)
Variable degraders
1mm Al - in
1mm Al - in
1mm Al - out
09.30 Detector biases; leakage currents OK - see attachments 3 & 4
FEE64 Temperatures OK - see attachment 5
good events; statistics OK - see attachment 6
No warnings/errors reported by merger since DAQ GO - see attachment 7
*all* system wide checks OK *except* ADC calibration of nnaida1, 21 and nnaida1 clock status (status=6)
09.50 layout 1-8 spectra - see attachments 8-17
Note pulser amplitude drifting for + polarity
*all* spectra zero'd
10.00 BigRIPS & BRIKEN DAQ stop
AIDA RIBF127R1/R1_114
Check ASIC control x2
10.03 AIDA RIBF127R1/R1_115
DAQ start BRIKEN 11
DAQ start BigRIPS 1011
11.12 Changing degraders (16 mm Al plate check?)
E17/F11 ambient temperature +24.7 deg C, RH 38.7%, d.p. +9.7 deg C
Julabo FL11006 set point +20 deg C, actual +20.0 deg C, water level OK (c. 70%)
HEC plot attached - see attach 8
11.17 AIDA RIBF127R1/R1_132
DAQ start BRIKEN 12
DAQ start BigRIPS 1012
Attempted to start new run. Lost beam. BigRIPS re-optimising
11.40 Found beam
Zero'd all histograms
Cleared online sorting histograms
AIDA RIBF127R1/R1_139
DAQ start BRIKEN 13
DAQ start BigRIPS 1013
12.14 Machine is down on AIDA run RIBF127R1/R1_145
HEC plot attached - see attach 9
Implants moved one detector upstream
12.24 Check ASIC Control x2
12.30 Beam back
12.31 AIDA RIBF127R1/R1_149
DAQ start BRIKEN 14
DAQ start BigRIPS 1014
12.35 Detector biases; leakage currents OK - see attachments 22 & 23
*online* analysis - ADC & disc data synchronised
analysis of R1_149 OK - see attachment 24 - deadtime <0.5%
12.39 Beam off RIBF127R1/R1_151
SRC cooling issues
13.55 Background run
AIDA RIBF127R1/R1_170
DAQ start BRIKEN 15, BigRIPS 1015
14.01 Beam still off
14.20 Stop background run
AIDA RIBF127R1/R1_175
15.30 Beam still off
DAQ continues OK
FEE64 temperatures OK - attach 24
Statistics OK - attach 25
Leakage currents OK - attach 26
No merger errors. No nnaida panics from USB console terminal.
|
|
Mon Nov 20 00:53:27 2017, TD, CB, Monday 20 November 9x
|
09.50 Detector biases & leakage currents OK - see attachments 1 & 2
FEE64 Temperatures OK - see attachment 3
DAQ statistics OK - see attachment 4
7x FEE64s <10k
4x FEE64s 10-20k
5x FEE64s 20-30k
3x FEE64s 30-40k
1x FEE64s 40-50k
4x FEE64 >50k (nnaida13 255k)
PLL & DCM Clock Lock Monitor OK - see attachment 5 & https://elog.ph.ed.ac.uk/AIDA/704 attachment 14
*all* system wide checks OK *except* nnaida7 fails ADC calibration
16.39 Detector biases & leakage currents OK - see attachments 5 & 6
FEE64 Temperatures OK - see attachment 7
DAQ statistics OK - see attachment 8
7x FEE64s <10k
4x FEE64s 10-20k
5x FEE64s 20-30k
3x FEE64s 30-40k
1x FEE64s 40-50k
4x FEE64 >50k (nnaida13 255k)
*all* system wide checks OK *except* nnaida7 fails ADC calibration |
|
Tue Nov 21 00:30:31 2017, TD, CB, Tuesday 21 November 20x
|
09.27 Detector biases & leakage currents OK - see attachments 1 & 2
FEE64 Temperatures OK - see attachment 3
DAQ statistics OK - see attachment 4
8x FEE64s <10k
2x FEE64s 10-20k
5x FEE64s 20-30k
3x FEE64s 30-40k
1x FEE64s 40-50k
5x FEE64 >50k (nnaida13 255k)
*all* system wide checks OK *except* nnaida7 fails ADC calibration
09.42 per FEE64 rate, 1.8.L, 1.8.W (20ms FSR) spectra - see attachments 5-10
all spectra zero'd
17.05 Detector biases & leakage currents OK - see attachments 11 & 12
FEE64 Temperatures OK - see attachment 13
DAQ statistics OK - see attachment 14
7x FEE64s <10k
4x FEE64s 10-20k
2x FEE64s 20-30k
4x FEE64s 30-40k
2x FEE64s 40-50k
5x FEE64 >50k (nnaida13 261k)
*all* system wide checks OK *except* nnaida7 fails ADC calibration
per FEE64 rate, 1.8.L, 1.8.W (20ms FSR) spectra - see attachments 15-20
pulser peak width nnaida17 69 ch FWHM, nnaida8 72 ch FWHM |
|
Thu Nov 23 00:41:43 2017, TD, CB, Thursday 23 November 10x
|
09.33 Detector biases & leakage currents OK - see attachments 1 & 2
FEE64 Temperatures OK - see attachment 3
DAQ statistics OK - see attachment 4
7x FEE64s <10k
4x FEE64s 10-20k
6x FEE64s 20-30k
2x FEE64s 30-40k
0x FEE64s 40-50k
5x FEE64 >50k (nnaida13 280k)
*all* system wide checks OK *except* nnaida7 fails ADC calibration
per FEE64 rate, 1.8.L, 1.8.W (20ms FSR) spectra - see attachments 5-10
pulser peak width nnaida17 70 ch FWHM, nnaida8 72 ch FWHM |
|
Thu Nov 23 01:10:43 2017, TD, CB, Detector bias & leakage currents - October & November 2017
|
https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=282765719 |
|
Thu Nov 23 01:12:33 2017, TD, CB, AIDA to safe shutdown state
|
10.11 Detector bias OFF
FEE64 power OFF
BNC PB-4 OFF
2x NIM bins OFF - fan tray OFF
Julabo FL11006 chiller OFF - 3ph mains switch OFF
USB ac mains relay OFF - flow interlock enabled - USB disconnected
Raspberry Pis ON
Tektronix AFG3102C ON |
|
Sun Feb 15 06:12:00 2015, TD, AEV & CG, How to mount USB disk with NTFS
|
Mount external USB expansion disk with NTFS filesystems
with the (superuser) command:
su root
password:
mount -t ntfs-3g /dev/sdd1 /ntfs
[root@aidas1 /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_aidas1-lv_root
50G 33G 15G 69% /
tmpfs 3.9G 4.3M 3.9G 1% /dev/shm
/dev/sda1 485M 61M 399M 14% /boot
/dev/mapper/vg_aidas1-lv_home
860G 201M 816G 1% /home
/dev/mapper/Data1-data10
1.8T 1.4T 309G 83% /data10
/dev/sdd1 3.7T 1.3T 2.4T 36% /ntfs |
|
Mon Feb 16 06:17:31 2015, TD, AEV & CG, Monday 16th Feb progress 10x
|
16/02/2015
1500: AIDA moved into operational position.
Water cooling circuit visually checked, fittings, etc for loose fittings/snags/kinks. No problems
Julabo FL11006 chiller moved into position with mains transformer, filled and turned on without problem (small leak from rear overflow, due to overfilling).
Took roughly 2 ~30L water containers to fill reservoir.
Brief leak at connection between back of chiller and yellow hose, bolt tightened and it stopped.
Having been turned on and loop filled, reservoir at ~50% full. Pressure ~10 psi.
No leaks observed in main coolant loop.
When standing downstream of AIDA RH flow gauge blurred due to high rotational speed, LH flow gauge rotating at ~1 rev/s.
Two FEE modules closest to centre in each block checked for power cabling, TS distribution and network cables. All ok.
MACBs front panel cabling checked, no obvious problems. TS distribution tree in order.
Power on for NIM bins and fan tray, network switches and Raspberry Pi. Not yet for the FEE module PSUs.
1 corner of each new USB power supply damaged in transit.
Fuses changed from 10A -> 16A in each PSU. Old 10A fuses taped to back of each module.
1730: Turned on nnaida1-16. Temperatures stable. All on firmware version 0xa4ed006.
Input pulser into all FEE cards. All FEEs bar nnaida5 well-behaving. nnaida12 has several missing channels.
Typical pulser peak width for -ve input modules was ~14ch FWHM with shaping time of 1us.
Typical pulser peak width for +ve input modules was ~27ch FWHM with shaping time of 1us.
Changing shaping time of nnaida10 from 1->2.5us reduced FWHM from 27ch to 23ch.
Attachment 1: temperatures & FEE firmware revisions nnaida1-16
|
|
Tue Jun 7 05:48:13 2016, TD, AE, ML & PJW, HEC spectra & implant profile
|
DSSSD#1 and DSSSD#6 ADC hit patterns (profile) and 1*H (ASIC#1 20GeV FSR) energy spectra |
|
Sun Jun 12 03:05:37 2016, TD, AE, CG, MA, Sunday 12 June 2016 part 2 11x
|
BigRIPS switched from 60Ti to 64V
AIDA48 (AIDA)2034 (EURICA) 2153 (BigRIPS)
start 08.44
stop 09.08 (FEE64 synchronisation lost -> power cycle & reboot FEE64s)
AIDA49 (AIDA) 2036 (EURICA) 2154 (BigRIPS)
Power cycle and reboot FEE64s
FADC power down (per PJCS recommendation)
All FEE64s pass all system wide checks *except* nnaida16 which failed the ADC calibration (not significant
as waveforms are disabled and FADC powered down anyway)
start 09.30
stop 09.58
ASIC settings 2016Jun12-09.09.37
AIDA50 (AIDA) 2037(EURICA) 2155 (BigRIPS)
start 10.02
stop 10.09 (beam lost)
ASIC settings 2016Jun12-10.10.08
changes to nnaida15 discriminator settings to reduce deadtime (pause-resume rate)
AIDA51 (AIDA) 2038 (EURICA) 2156 (BigRIPS)
start 10.45
stop 11.55
ASIC settings 2016Jun12-10.10.08
aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA51_0
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 163625493 ( 974305.9 Hz)
Other data format: 97494507 ( 580529.8 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1227 ( 7.3 Hz)
RESUME: 1227 ( 7.3 Hz)
SYNC100: 64065 ( 381.5 Hz)
FEE64 disc: 96631002 ( 575388.1 Hz)
MBS info: 796986 ( 4745.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 213982 ( 1274.2 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 167.941 s
FEE module #: 1 elapsed dead time 0.142 s
FEE module #: 2 elapsed dead time 1.125 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.307 s
FEE module #: 6 elapsed dead time 0.579 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.082 s
FEE module #: 10 elapsed dead time 0.112 s
FEE module #: 11 elapsed dead time 3.204 s
FEE module #: 12 elapsed dead time 0.005 s
FEE module #: 13 elapsed dead time 0.212 s
FEE module #: 14 elapsed dead time 0.426 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.000 s
FEE module #: 18 elapsed dead time 0.046 s
FEE module #: 19 elapsed dead time 0.000 s
FEE module #: 20 elapsed dead time 0.064 s
FEE module #: 21 elapsed dead time 0.319 s
FEE module #: 22 elapsed dead time 0.248 s
FEE module #: 23 elapsed dead time 0.010 s
FEE module #: 24 elapsed dead time 0.190 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 60.988s ( 524.691 blocks/s, 32.793 Mb/s)
See attachments 1-7 for AIDA51 rates etc
See attachment 8 for FEE64 synchronisation check (file AIDA51_0) - OK
AIDA52 (AIDA) 2039 (EURICA) 2158 (BigRIPS)
Start 11.56
Stop 12.54 (AIDA FEE lose sync)
Settings file 2016Jun12-10.10.08
Checked correlation of FEEs in file AIDA52_0 at 12.06 and everything seems ok.
aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA52_9
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 163879044 ( 985540.7 Hz)
Other data format: 97240956 ( 584790.6 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1315 ( 7.9 Hz)
RESUME: 1315 ( 7.9 Hz)
SYNC100: 63433 ( 381.5 Hz)
FEE64 disc: 96378426 ( 579603.4 Hz)
MBS info: 796467 ( 4789.8 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 215707 ( 1297.2 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 1 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 166.283 s
FEE module #: 1 elapsed dead time 0.088 s
FEE module #: 2 elapsed dead time 0.724 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.510 s
FEE module #: 6 elapsed dead time 1.337 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.087 s
FEE module #: 10 elapsed dead time 0.121 s
FEE module #: 11 elapsed dead time 2.706 s
FEE module #: 12 elapsed dead time 0.000 s
FEE module #: 13 elapsed dead time 0.127 s
FEE module #: 14 elapsed dead time 0.191 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.000 s
FEE module #: 18 elapsed dead time 0.058 s
FEE module #: 19 elapsed dead time 0.000 s
FEE module #: 20 elapsed dead time 0.071 s
FEE module #: 21 elapsed dead time 0.399 s
FEE module #: 22 elapsed dead time 0.222 s
FEE module #: 23 elapsed dead time 0.015 s
FEE module #: 24 elapsed dead time 0.061 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 47.215s ( 677.753 blocks/s, 42.360 Mb/s)
12.20 FEE64 temperatures - see attachment 9
Note with all FADC powered off all temperatures now green cf. attachment 8 https://elog.ph.ed.ac.uk/AIDA/305
Julabo FL11006 set point +17 deg C (B3F ambient temperature +25.1 deg C, d.p. +11.2 deg C, RH 40.3%)
13.11 AIDA52_13 per FEE64 ADC timestamp activity OK
12.54 AIDA FEE64 lose synchronisation
No change in system-wide checks
AIDA53 (AIDA) 2040 (EURICA) 2159 (BigRIPS)
Power cycled and rebooted FEE64s
System-wide checks all OK except 6 failures for ADC calibration - not retried
Start 13.09
Stop 13.44 (stop to reset EURICA VME crate)
ASIC Settings file 2016Jun12-10.10.08
13.17 AIDA53_0 FEE64 sync check - OK
aidas1> ~/td/GREAT/analyser2 /TapeData/NP1512/AIDA53_0
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 167386044 ( 1100597.1 Hz)
Other data format: 93733956 ( 616319.7 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 714 ( 4.7 Hz)
RESUME: 714 ( 4.7 Hz)
SYNC100: 58017 ( 381.5 Hz)
FEE64 disc: 92936706 ( 611077.6 Hz)
MBS info: 737805 ( 4851.2 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 166106 ( 1092.2 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 1 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 152.087 s
FEE module #: 1 elapsed dead time 0.020 s
FEE module #: 2 elapsed dead time 0.209 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.107 s
FEE module #: 6 elapsed dead time 0.824 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.008 s
FEE module #: 9 elapsed dead time 0.003 s
FEE module #: 10 elapsed dead time 0.000 s
FEE module #: 11 elapsed dead time 0.378 s
FEE module #: 12 elapsed dead time 0.017 s
FEE module #: 13 elapsed dead time 0.000 s
FEE module #: 14 elapsed dead time 0.059 s
FEE module #: 15 elapsed dead time 0.172 s
FEE module #: 16 elapsed dead time 1.101 s
FEE module #: 17 elapsed dead time 0.000 s
FEE module #: 18 elapsed dead time 0.018 s
FEE module #: 19 elapsed dead time 0.000 s
FEE module #: 20 elapsed dead time 0.093 s
FEE module #: 21 elapsed dead time 0.216 s
FEE module #: 22 elapsed dead time 0.161 s
FEE module #: 23 elapsed dead time 0.665 s
FEE module #: 24 elapsed dead time 0.201 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 25.180s ( 1270.866 blocks/s, 79.429 Mb/s)
AIDA54 (AIDA) 2041 (EURICA) 2160 (BigRIPS)
Start 13.46
Stop 13.53 (loss of sync AIDA54_0)
ASIC Settings file 2016Jun12-10.10.08
System-wide checks OK following loss of sync
19:25
Stable operation most of the afternoon through multiple DAQ start/stops.
FEEs remained synchronised throughout.
SYNCs lost in run AIDA57_37 leading to run stop/start.
-> in AIDA57_37 noticed relatively large chunks of data lost in nnaida6. (see attachments 10+11
19.40 Cumulative pause-resume time for AIDA57_37 (see above) and most recent
complete data file. The pause-resume times are relatively large compared to
other FEE64s but low relative to the measurement time. But we should remember
that we have observed periods of inactivity not bounded by pause-resume tags.
aidas1> ./analyser2 /TapeData/NP1512/AIDA57_37
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 164401851 ( 929891.0 Hz)
Other data format: 96718149 ( 547057.9 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1081 ( 6.1 Hz)
RESUME: 1081 ( 6.1 Hz)
SYNC100: 67440 ( 381.5 Hz)
FEE64 disc: 95842999 ( 542107.9 Hz)
MBS info: 805548 ( 4556.3 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 194112 ( 1097.9 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 176.797 s
FEE module #: 1 elapsed dead time 0.149 s
FEE module #: 2 elapsed dead time 0.741 s
FEE module #: 3 elapsed dead time 0.004 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.425 s
FEE module #: 6 elapsed dead time 1.697 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.118 s
FEE module #: 10 elapsed dead time 0.084 s
FEE module #: 11 elapsed dead time 1.064 s
FEE module #: 12 elapsed dead time 0.014 s
FEE module #: 13 elapsed dead time 0.063 s
FEE module #: 14 elapsed dead time 0.267 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.000 s
FEE module #: 18 elapsed dead time 0.010 s
FEE module #: 19 elapsed dead time 0.000 s
FEE module #: 20 elapsed dead time 0.059 s
FEE module #: 21 elapsed dead time 0.125 s
FEE module #: 22 elapsed dead time 0.225 s
FEE module #: 23 elapsed dead time 0.000 s
FEE module #: 24 elapsed dead time 0.205 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 23.047s ( 1388.475 blocks/s, 86.780 Mb/s)
aidas1> ./analyser2 /TapeData/NP1512/AIDA58_6
*** GREAT format 3.2.0 analyser - TD - May 2014
*** ADC data: block: 14849 ptr: 14685 data: 0xC40181C9 module: 16 fail: 0 range: 0 id: 1025 channel:
1 adc: 33225 ts: 0x01AB000000A0 OLD: ts: 0x01AB1000003E tag: 3 type: 0
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 160685457 ( 937303.1 Hz)
Other data format: 100434543 ( 585850.2 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1282 ( 7.5 Hz)
RESUME: 1282 ( 7.5 Hz)
SYNC100: 65397 ( 381.5 Hz)
FEE64 disc: 99536560 ( 580612.1 Hz)
MBS info: 830022 ( 4841.6 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 213213 ( 1243.7 Hz)
Timewarps: ADC: 1 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
Timestamp elapsed time: 171.434 s
FEE module #: 1 elapsed dead time 0.091 s
FEE module #: 2 elapsed dead time 0.870 s
FEE module #: 3 elapsed dead time 0.000 s
FEE module #: 4 elapsed dead time 0.000 s
FEE module #: 5 elapsed dead time 0.107 s
FEE module #: 6 elapsed dead time 0.518 s
FEE module #: 7 elapsed dead time 0.000 s
FEE module #: 8 elapsed dead time 0.000 s
FEE module #: 9 elapsed dead time 0.058 s
FEE module #: 10 elapsed dead time 0.095 s
FEE module #: 11 elapsed dead time 2.397 s
FEE module #: 12 elapsed dead time 0.055 s
FEE module #: 13 elapsed dead time 0.157 s
FEE module #: 14 elapsed dead time 0.238 s
FEE module #: 15 elapsed dead time 0.000 s
FEE module #: 16 elapsed dead time 0.000 s
FEE module #: 17 elapsed dead time 0.000 s
FEE module #: 18 elapsed dead time 0.013 s
FEE module #: 19 elapsed dead time 0.000 s
FEE module #: 20 elapsed dead time 0.174 s
FEE module #: 21 elapsed dead time 0.298 s
FEE module #: 22 elapsed dead time 0.280 s
FEE module #: 23 elapsed dead time 0.004 s
FEE module #: 24 elapsed dead time 0.071 s
FEE module #: 25 elapsed dead time 0.000 s
FEE module #: 26 elapsed dead time 0.000 s
FEE module #: 27 elapsed dead time 0.000 s
FEE module #: 28 elapsed dead time 0.000 s
FEE module #: 29 elapsed dead time 0.000 s
FEE module #: 30 elapsed dead time 0.000 s
FEE module #: 31 elapsed dead time 0.000 s
FEE module #: 32 elapsed dead time 0.000 s
*** Program elapsed time: 18.234s ( 1754.927 blocks/s, 109.683 Mb/s) |
|
Sat Jun 4 04:53:25 2016, TD, AE, CG, DK, GL YL & PJW (Team AIDA), Saturday 4 June 2016 8x
|
12.27 DAQ run R61 started 01.30
Appears to have stopped abruptly at ~06.55
All FEE64 AIDA SYNCs zero
Attempted normal halt to run - DAQ STOP, toggle merge pause, stop tape server - OK
Attempted normal start to run - start tape server, DAQ GO, toggle merge pause - OK
Note during DAQ GO briefly obtained 'unable to connect' error from nnaida1 which resolved
itself a few seconds later
Atachment 1 - AIDA SYNCs
2 - MERGE and server terminal sessions
3 - TapeServer
4 - R61 disk files
5 - Detector biases and leakage currents - OK
Leakage current has decreased since yesterday. B2F/F11 ambient temperature
~25 deg C but EURICA now closed.
6 - AIDA discriminators
7 - AIDA good events
From the MERGE server terminal session ...
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Timestamp (4d087c824ee)[Q 5, Stream 5] is older than previous (4d097c800a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 4d097c800a1 or greater
Merge: Last sync pulse time: 4d097c400a0
Merge: Last timestamp seen: 4d097c800a0
Merge: Restarting at timestamp 4d097cc00a0
Merge: Timestamp (4e122b40d1e)[Q 5, Stream 5] is older than previous (4e132b400a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 4e132b400a1 or greater
Merge: Last sync pulse time: 4e132b000a0
Merge: Last timestamp seen: 4e132b400a0
Merge: Restarting at timestamp 4e132b800a0
Merge: Timestamp (4e1662818a6)[Q 5, Stream 5] is older than previous (4e1762800a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 4e1762800a1 or greater
Merge: Last sync pulse time: 4e1762400a0
Merge: Last timestamp seen: 4e1762800a0
Merge: Restarting at timestamp 4e1762c00a0
Merge: Timestamp (4e1b89803c6)[Q 5, Stream 5] is older than previous (4e1c89800a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 4e1c89800a1 or greater
Merge: Last sync pulse time: 4e1c89400a0
Merge: Last timestamp seen: 4e1c89800a0
Merge: Restarting at timestamp 4e1c89c00a0
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Trying to add items to Q index 7 (Link 7) but queue already full
Merge: Timestamp (4f2580400c6)[Q 5, Stream 5] is older than previous (4f2680400a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 4f2680400a1 or greater
Merge: Last sync pulse time: 4f2680000a0
Merge: Last timestamp seen: 4f2680400a0
Merge: Restarting at timestamp 4f2680800a0
Merge: Timestamp (5026a041c06)[Q 5, Stream 5] is older than previous (5027a0400a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 5027a0400a1 or greater
Merge: Last sync pulse time: 5027a0000a0
Merge: Last timestamp seen: 5027a0400a0
Merge: Restarting at timestamp 5027a0800a0
Merge: Timestamp (5083c7008c6)[Q 5, Stream 5] is older than previous (5084c7000a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 5084c7000a1 or greater
Merge: Last sync pulse time: 5084c6c00a0
Merge: Last timestamp seen: 5084c7000a0
Merge: Restarting at timestamp 5084c7400a0
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Trying to add items to Q index 2 (Link 2) but queue already full
Merge: Timestamp (523c5c80b4e)[Q 5, Stream 5] is older than previous (523d5c800a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 523d5c800a1 or greater
Merge: Last sync pulse time: 523d5c400a0
Merge: Last timestamp seen: 523d5c800a0
Merge: Restarting at timestamp 523d5cc00a0
Merge: Timestamp (527ddc03916)[Q 5, Stream 5] is older than previous (527edc000a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 527edc000a1 or greater
Merge: Last sync pulse time: 527edbc00a0
Merge: Last timestamp seen: 527edc000a0
Merge: Restarting at timestamp 527edc400a0
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Trying to add items to Q index 4 (Link 4) but queue already full
Merge: Timestamp (52e9c2c0456)[Q 5, Stream 5] is older than previous (52eac2c00a0) [Q 5, Stream 5]
Merge: Attempting resynchronisation to timestamp 52eac2c00a1 or greater
Merge: Last sync pulse time: 52eac2800a0
Merge: Last timestamp seen: 52eac2c00a0
Merge: Restarting at timestamp 52eac3000a0
Queue full messages do not appear to be correlated (currently at least) with the busiest FEE64s
12.50 AIDA trigger to EURICA OR(DSSSD#1 - #5) .AND.NOT. DSSSD#6 ~1.8kHz
.AND.EURICA ~50Hz (no significant change in rate when DSSSD#6 veto removed)
15.30 DAQ found stopped - AIDA SYNCs zero - data transfer disabled, histogramming enabled
System wide checks
nnaida19 fails 'ASIC clock timestamp' and 'SYNC errors not counting tests'
all other FEE64s pass all tests |
|