|
|
|
AIDA
GELINA
BRIKEN
nToF
CRIB
ISOLDE
CIRCE
nTOFCapture
DESPEC
DTAS
EDI_PSA
179Ta
CARME
StellarModelling
DCF
K40
|
| DESPEC |
 |
|
|
|
Message ID: 187
Entry time: Wed Mar 10 07:16:02 2021
|
| Author: |
OH, LS |
| Subject: |
Wednesday 10th March 08:00-24:00 |
|
|
08:16 System wide checks all ok
Statistics - attachment 1
Temperatures - attachment 2
bias - attachment 3
Merger running 4.5e6 events per second
Tape data writing at 42MB per second
Currently no beam
09:45 While there is no beam we will perform a longer test of the new merger.
Run R20 stopped and merger changed to neew version
Started R21.
The now usual behaviour of additional files at start of merge observed.
N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was
set to going
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_0
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_1
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_2
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_3
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_4
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_5
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_6
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_7
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_8
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_9
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_10
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_11
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_12
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_13
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_14
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_15
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_16
-rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17
-rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18
TapeData rate of 13280 kB/sec
No errors observed in ucesb
Correlations seen between time machine in AIDA and Ge in online
Pulser rate in the online makes sense
With the current data rate remaining HDD space on current drive will last around 89 hours.
09.20 analysis of file R21_27
ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data
10:24 Message while performing system wide checks:
Get returned with an error
error: SOAP http transport timed out after 10000 ms
NONE
error: SOAP http transport timed out after 10000 ms
while executing
"$transport $procVarName $url $req"
(procedure "::SOAP::invoke" line 18)
invoked from within
"::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10"
("eval" body line 1)
invoked from within
"eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args"
(procedure "XAIDAAccessClient__Get" line 1)
invoked from within
"XAIDAAccessClient__Get $Addr"
aida02 restarted itself during the reset process
Looking at the log messages on aida02 cannot see any reason for the cause of the restart.
System wide checks following the restart all ok.
When restarting the MBS relay errors observed until a timestamp was observed:
Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000
10:55 Statistics - attachment 5
Temperature - attachment 6
Bias - attachment 7
11:03 Time machine correlation spectra with new merger
AIDA - FATIMA - Attachment 8
AIDA - Ge - Attachment 9
11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the
layout to determine which FEE/ASIC caused the events before checking ASIC control
12:12 System wide checks all ok *except adc calibration which is same as before
Statistics - attachment 10
Temperature - attachment 11
Bias and leakage currents ok - attachment 12
A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat.
After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time.
They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon
cable from the VME scalers.
Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of
DSSD3.
12:50 We have waveforms for some FEES
Layout 7 attachment -14
Layout 8 - attachment 15
14.00 (LS)
Still no beam
System wide checks okay except same as before:
**FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module**
Statistics (attachment16)
Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16):
aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up
aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on
FEE Temps (attachment17)
Leakage currents written to sheets(attachment18)
Merger ~ 45M items/s
TapeServer~ 14MB/s
14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file
R22_98, rate spectra attached (attachment19)
14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw
no timewarps so we are not losing anything.
Restarted MBS relay, errors have not reappeared so far
16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
Statistics (attachment20)
Rate spectra (attachment21)
FEE Temps (attachment22)
Leakage currents written to sheets(attachment23)
Merger ~ 4.3M items/s
TapeServer~ 14MB/s
Current file R22_148
16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from
S452f115
Around R22_150
18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm)
Rates of R22_122 (attachment25):
First DSSSD (fee0-fee3) ~66/s
Second DSSSD (fee4-fee7) ~49/s
Third DSSSD (fee8-fee11) ~22/s
Rates of R22_178 (attachment26):
First DSSSD (fee0-fee3) ~122/s
Second DSSSD (fee4-fee7) ~94/s
Third DSSSD (fee8-fee11) ~49/s
18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
Statistics (attachment27)
Rate spectra (attachment28)
FEE Temps (attachment29)
Leakage currents written to sheets(attachment30), look to be on the way down
Merger ~ 4.5M items/s
TapeServer~ 15MB/s
18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier
19:34 MBS DAQ Crashed
20:00 DAQ is still down. They have Sultan working on it.
20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre
reboot)
We are borrowing a HDD from the SHIP group
20:30 System wide checks all ok
Statistics - attachment 31
Temperature - attachment 32
Bias - attachment 33
22:10 MBS DAQ has crashed
Merger terminal is now showing a large number of bad merge events
It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS.
22:42 System wide checks all ok
Statistics - attachment 34
Temperature - Attachment 35
Bias and leakage currents ok - attachment 36
23:33 System wide checks all ok - attachment 37
Temperatures - attachment 38
Bias and leakage currents ok - attachment 39 |
|
|
|
|
|
|
|
*** TDR format 3.3.0 analyser - TD - January 2019
*** ERROR: READ I/O error: 5002
blocks: 32000
ADC data format: 259319480 ( 1687258.2 Hz)
Other data format: 2600520 ( 16920.2 Hz)
Sample trace data format: 0 ( 0.0 Hz)
Undefined format: 0 ( 0.0 Hz)
Other data format type: PAUSE: 1272 ( 8.3 Hz)
RESUME: 1272 ( 8.3 Hz)
SYNC100: 573 ( 3.7 Hz)
WR48-63: 573 ( 3.7 Hz)
FEE64 disc: 302889 ( 1970.7 Hz)
MBS info: 2293941 ( 14925.5 Hz)
Other info: 0 ( 0.0 Hz)
ADC data range bit set: 0 ( 0.0 Hz)
Timewarps: ADC: 0 ( 0.0 Hz)
PAUSE: 0 ( 0.0 Hz)
RESUME: 0 ( 0.0 Hz)
SYNC100: 0 ( 0.0 Hz)
WR48-63: 0 ( 0.0 Hz)
FEE64 disc: 0 ( 0.0 Hz)
MBS info: 0 ( 0.0 Hz)
Undefined: 0 ( 0.0 Hz)
Sample trace: 0 ( 0.0 Hz)
*** Timestamp elapsed time: 153.693 s
FEE elapsed dead time(s) elapsed idle time(s)
1 0.000 150.055
2 0.129 153.277
3 1.695 152.203
4 0.000 97.174
5 15.630 152.471
6 0.533 122.407
7 0.048 144.955
8 17.221 151.934
9 26.571 143.076
10 0.063 148.982
11 11.592 141.197
12 0.000 0.000
13 0.000 0.000
14 0.000 0.000
15 0.000 0.000
16 0.000 0.000
17 0.000 0.000
18 0.000 0.000
19 0.000 0.000
20 0.000 0.000
21 0.000 0.000
22 0.000 0.000
23 0.000 0.000
24 0.000 0.000
25 0.000 0.000
26 0.000 0.000
27 0.000 0.000
28 0.000 0.000
29 0.000 0.000
30 0.000 0.000
31 0.000 0.000
32 0.000 0.000
*** Statistics
FEE ADC Data Other Data Sample Undefined Pause Resume SYNC100 WR48-63 Disc MBS Other HEC Data
0 20482535 1736 0 0 381 381 52 52 0 870 0 0
1 26566394 1030 0 0 0 0 56 56 0 918 0 0
2 26980761 1455064 0 0 2 2 87 87 302889 1151997 0 0
3 26884821 1140284 0 0 5 5 65 65 0 1140144 0 0
4 7516935 23 0 0 0 0 7 7 0 9 0 0
5 16098094 1219 0 0 577 577 31 31 0 3 0 0
6 9441595 40 0 0 2 2 18 18 0 0 0 0
7 8769482 38 0 0 2 2 17 17 0 0 0 0
8 32618578 336 0 0 98 98 70 70 0 0 0 0
9 32790396 426 0 0 136 136 77 77 0 0 0 0
10 16087963 82 0 0 1 1 40 40 0 0 0 0
11 35081926 242 0 0 68 68 53 53 0 0 0 0
12 0 0 0 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0 0 0 0
*** Timewarps
FEE ADC Pause Resume SYNC100 WR48-63 Disc MBS Undefined Samples
0 0 0 0 0 0 0 0 0 0
1 0 0 0 0 0 0 0 0 0
2 0 0 0 0 0 0 0 0 0
3 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 0 0 0 0
5 0 0 0 0 0 0 0 0 0
6 0 0 0 0 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0
8 0 0 0 0 0 0 0 0 0
9 0 0 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0 0 0
11 0 0 0 0 0 0 0 0 0
12 0 0 0 0 0 0 0 0 0
13 0 0 0 0 0 0 0 0 0
14 0 0 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0 0 0
16 0 0 0 0 0 0 0 0 0
17 0 0 0 0 0 0 0 0 0
18 0 0 0 0 0 0 0 0 0
19 0 0 0 0 0 0 0 0 0
20 0 0 0 0 0 0 0 0 0
21 0 0 0 0 0 0 0 0 0
22 0 0 0 0 0 0 0 0 0
23 0 0 0 0 0 0 0 0 0
24 0 0 0 0 0 0 0 0 0
25 0 0 0 0 0 0 0 0 0
26 0 0 0 0 0 0 0 0 0
27 0 0 0 0 0 0 0 0 0
28 0 0 0 0 0 0 0 0 0
29 0 0 0 0 0 0 0 0 0
30 0 0 0 0 0 0 0 0 0
31 0 0 0 0 0 0 0 0 0
32 0 0 0 0 0 0 0 0 0
*** Program elapsed time:37059.496s ( 0.863 blocks/s, 0.054 Mb/s)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|