|
|
|
AIDA
GELINA
BRIKEN
nToF
CRIB
ISOLDE
CIRCE
nTOFCapture
DESPEC
DTAS
EDI_PSA
179Ta
CARME
StellarModelling
DCF
K40
|
| DESPEC |
 |
|
|
|
Message ID: 140
Entry time: Tue Mar 10 23:17:31 2020
|
| Author: |
DK, LS, PW, TD, CA |
| Subject: |
Wednesday 11 March 2020 |
|
|
00:15 Shift change near AIDA R6_475
We continue with spill cycles around 0.2 Hz or one beginning once each ~5 seconds.
All system-wide checks passed successfully
Temperatures ok (aida01 still a bit hot) - see attachment #1
Stats as normal (SSD3 still noisy) - see attachment #2
Bias and pulser settings remain sane - see attachment #3
Writing data to disk at ~26 MB/s as yesterday - see attachment #4
Merging fine, 2.6 to 3.3 Mega items / sec (all aida## seen to go on) - see attachment #5
Rates look not unreasonable for expected implantation in short time window - see attachment #6
2:26
We are near R6_574
All system checks are fine
Temps are fine - attachment #7
Stats look as before - attachment #8
Biases and leaks typical - attachment #9
Merger fine, 3 x 10^6 items / sec (skip screencap)
TapeServer still sits near 25 MB/s (skip screencap)
Cumulative implants look reasonable, but NB that histos have not been cleared - attachment #10
2:44 R6_587 or R6_588
The spills seem to have stopped based on the beeping scalars. Confirmed with other group there are no scint counts.
See attachment of AIDA rates where no implants can be seen. See attachment #11
2:46 Spill beeps heard briefly, but they cut again. R6_590
2:49 Beam seems to be back R6_592
Scints have counts in DESPEC DAQ, scalars are beeping, AIDA sees implants. No problem. See attachment #12 for implants.
4:35
System checks all pass (except expected error on master clock)
Temps okay - attachment #13
Biases and leak currents normal - attachment #14
Stats as before - attachment #15
Merger going, 3 mega items per sec
TapeServer going, 25 MB/s
5:59 Beam is gone again, near R6_738. As before, we get consistent confirmation from the trigger clicks and the plastic scint data of DESPEC group.
See attachment #16 which shows no implants in the rate.
Contact phones for Control Room: 2222
We called, they are aware of a problem with the magnets and they are working to resolve it. So we will await them.
Expert is scheduled for 7:30, but he may come earlier like 7:00 if he can make it in.
We leave AIDA writing to disk since there are some longer lived daughters (hour or more) so we may still get some interesting decays with beam off
7:14 Beam is still off, but I make some notes
System wide checks all okay
Temps as normal (see attachment #17)
Biases and leak currents okay (see attachment #18)
Stats as normal (see attachment #19)
Merger going at 3 mega cps
TapeServer on, around 25 MB/sec
Implants not seen in histogram rate as expected (see attachment #20)
7:32 We called the operators to confirm that they don't change anything before calling us first.
Surely they said they don't touch the settings without telling us first, and just one magnet (maybe upstream of SIS) is down, and no plan to alter settings.
8.00 shift change at file R6_823
beam is back on, no settings changed.
particles per spill is similar to before (2k~3k per spill)
implants now being seen in rate histogram (attachment 21)
8.32 system wide checks all okay
FEE temperatures are normal (attachment 22)
leak currents are okay and have been recorded to sheet (attachment 23)
good event stats seem normal, although noted that all FEEs in DSSD3 now over 100k (attachment 24)
merger rate is running around 3M items/sec
tape service rate is running around 26MB/sec
9.20 high energy spectra histograms for even FEEs was not displaying correctly (attachment 25). can also be seen on yesterdays elog on attachment 34
problem can be seen clearly in /tmp/LayOut6.mlf for the line corresponding to aida4 was 1.8h not 1.8H as the information was somehow saved in the layout (attachment 26).
remaking each histogram and saving layout6 seems to be fixed (attachment 27)
Top of the error message showed the problem, like:
====
Using /tmp/LayOut6.mlf
restore server=aida02 spectrum=1.8.H to gallery=1 overlap=1
restore server=aida04 spectrum=1.8.h to gallery=2 overlap=1
SpecDetails returned with an error
0x30004
SOAP-ENV:Server-Error detected in method SpecDetails
Invalid spectrum name
invoked from within
"SpecDetails 1.8.h"
(in namespace inscope "::urn:SpectrumService" script line 1)
invoked from within
====
10.05 beam is being stopped to change some thresholds
10.34 system wide checks all okay
FEE temperatures are normal (attachment 28)
leak currents are okay and have been recorded to spreadsheet (attachment 29)
good event stats are normal, as previously mentioned aida11 rate was over 100k this looks to have just been a fluctuation (attachment 30)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 24MB/sec
implants not seen in histogram rate as expected as beam still down (see attachment 30)
10.42 beam still down but taking data, currently file finishing R6_946
as of now we have 3.2Tb left which should correspond to around 26 hours left until space is full
have zeroed all histograms so will take screenshots at 11.00
11.06 checked rates histograms, for aida3 some low energy channels in ASICs 1,2, and 3 were missing so performed ASIC control checks
11.25 beam back file S480/R6_975
12.00 analysis file S480/R6_1001 - attachment 33
FEE aida09 deadtime c. 5% (highest rate good events c. 250k), all other FEE64s << 1%
all HEC data rate c. 1kHz
all ADC data rate c. 1.2MHz
12.23 system wide checks all okay
FEE temperatures are normal (attachment 34)
leak currents are okay and have been recorded to spreadsheet (attachment 35)
good event stats are okay (attachment 36)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 26MB/sec
13.00 beam off, unknown reason at the moment, but can confirm from rates histogram (attachment 37)
seems to be intermittent. file R6_1051
13.10 beam back on, no one was called. currently on file R6_1061
13.22 beam off as going to higher intensity beam. file R6_1069
14.42 system wide checks all okay
FEE temperatures are normal (attachment 38)
leak currents are okay and have been recorded to spreadsheet (attachment 39)
good event stats are okay (attachment 40)
merger is running at a rate of 3M items/sec
tape service is running at a rate of 24M/sec
14.48 on file R6_1131
16.19 beam stop for 2 hours. TapeServer stopped and restarted - AIDA writing to No Storage
file R6_1193
16:35 compression of files R6_[200-999] commenced (nice -20)
16:40 system wide checks all okay
FEE temperatures are normal (attachment 41)
leak currents are okay and have been recorded to spreadsheet (attachment 42)
good event stats are okay (attachment 43) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
*** TDR format 3.3.0 analyser - TD - January 2019
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 87080536 ( 1162937.9 Hz)
Other data format: 174903340 ( 2335788.5 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 31 ( 0.4 Hz)
RESUME: 31 ( 0.4 Hz)
SYNC100: 87327938 ( 1166241.9 Hz)
WR48-63: 87327938 ( 1166241.9 Hz)
FEE64 disc: 247402 ( 3304.0 Hz)
MBS info: 0 ( 0.0 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 73144 ( 976.8 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 74.880 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 0.000
2 0.000 0.000
3 0.144 0.000
4 0.000 0.000
5 0.033 0.000
6 0.000 0.000
7 0.033 0.000
8 4.160 3.444
9 0.111 0.000
10 0.116 0.000
11 0.111 0.000
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 2537909 5321817 0 0 1 1 2619908 2619908 81999 0 0 13572
1 12374388 24792837 0 0 0 0 12389075 12389075 14687 0 0 7144
2 5995747 12118373 0 0 0 0 6038040 6038040 42293 0 0 8968
3 3245929 6571235 0 0 3 3 3272386 3272386 26457 0 0 13205
4 1018810 2094536 0 0 0 0 1037782 1037782 18972 0 0 5843
5 3175371 6374045 0 0 1 1 3183138 3183138 7767 0 0 3654
6 1851746 3764473 0 0 0 0 1872073 1872073 20327 0 0 9037
7 2503081 5045008 0 0 1 1 2516029 2516029 12948 0 0 6580
8 19164914 38353829 0 0 20 20 19172901 19172901 7987 0 0 1389
9 15009332 30028700 0 0 2 2 15012676 15012676 3344 0 0 894
10 5573107 11166435 0 0 2 2 5579846 5579846 6739 0 0 1608
11 14630202 29272052 0 0 1 1 14634084 14634084 3882 0 0 1250
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time:43185.973s ( 0.741 blocks/s, 0.046 Mb/s)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|