|
AIDA
GELINA
BRIKEN
nToF
CRIB
ISOLDE
CIRCE
nTOFCapture
DESPEC
DTAS
EDI_PSA
179Ta
CARME
StellarModelling
DCF
K40
|
DESPEC |
 |
|
Message ID: 187
Entry time: Wed Mar 10 07:16:02 2021
|
Author: |
OH, LS |
Subject: |
Wednesday 10th March 08:00-24:00 |
|
|
08:16 System wide checks all ok
Statistics - attachment 1
Temperatures - attachment 2
bias - attachment 3
Merger running 4.5e6 events per second
Tape data writing at 42MB per second
Currently no beam
09:45 While there is no beam we will perform a longer test of the new merger.
Run R20 stopped and merger changed to neew version
Started R21.
The now usual behaviour of additional files at start of merge observed.
N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was
set to going
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_0
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_1
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_2
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_3
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_4
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_5
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_6
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_7
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_8
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_9
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_10
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_11
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_12
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_13
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_14
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_15
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_16
-rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17
-rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18
TapeData rate of 13280 kB/sec
No errors observed in ucesb
Correlations seen between time machine in AIDA and Ge in online
Pulser rate in the online makes sense
With the current data rate remaining HDD space on current drive will last around 89 hours.
09.20 analysis of file R21_27
ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data
10:24 Message while performing system wide checks:
Get returned with an error
error: SOAP http transport timed out after 10000 ms
NONE
error: SOAP http transport timed out after 10000 ms
while executing
"$transport $procVarName $url $req"
(procedure "::SOAP::invoke" line 18)
invoked from within
"::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10"
("eval" body line 1)
invoked from within
"eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args"
(procedure "XAIDAAccessClient__Get" line 1)
invoked from within
"XAIDAAccessClient__Get $Addr"
aida02 restarted itself during the reset process
Looking at the log messages on aida02 cannot see any reason for the cause of the restart.
System wide checks following the restart all ok.
When restarting the MBS relay errors observed until a timestamp was observed:
Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000
10:55 Statistics - attachment 5
Temperature - attachment 6
Bias - attachment 7
11:03 Time machine correlation spectra with new merger
AIDA - FATIMA - Attachment 8
AIDA - Ge - Attachment 9
11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the
layout to determine which FEE/ASIC caused the events before checking ASIC control
12:12 System wide checks all ok *except adc calibration which is same as before
Statistics - attachment 10
Temperature - attachment 11
Bias and leakage currents ok - attachment 12
A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat.
After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time.
They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon
cable from the VME scalers.
Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of
DSSD3.
12:50 We have waveforms for some FEES
Layout 7 attachment -14
Layout 8 - attachment 15
14.00 (LS)
Still no beam
System wide checks okay except same as before:
**FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module**
Statistics (attachment16)
Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16):
aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up
aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on
FEE Temps (attachment17)
Leakage currents written to sheets(attachment18)
Merger ~ 45M items/s
TapeServer~ 14MB/s
14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file
R22_98, rate spectra attached (attachment19)
14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw
no timewarps so we are not losing anything.
Restarted MBS relay, errors have not reappeared so far
16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
Statistics (attachment20)
Rate spectra (attachment21)
FEE Temps (attachment22)
Leakage currents written to sheets(attachment23)
Merger ~ 4.3M items/s
TapeServer~ 14MB/s
Current file R22_148
16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from
S452f115
Around R22_150
18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm)
Rates of R22_122 (attachment25):
First DSSSD (fee0-fee3) ~66/s
Second DSSSD (fee4-fee7) ~49/s
Third DSSSD (fee8-fee11) ~22/s
Rates of R22_178 (attachment26):
First DSSSD (fee0-fee3) ~122/s
Second DSSSD (fee4-fee7) ~94/s
Third DSSSD (fee8-fee11) ~49/s
18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
Statistics (attachment27)
Rate spectra (attachment28)
FEE Temps (attachment29)
Leakage currents written to sheets(attachment30), look to be on the way down
Merger ~ 4.5M items/s
TapeServer~ 15MB/s
18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier
19:34 MBS DAQ Crashed
20:00 DAQ is still down. They have Sultan working on it.
20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre
reboot)
We are borrowing a HDD from the SHIP group
20:30 System wide checks all ok
Statistics - attachment 31
Temperature - attachment 32
Bias - attachment 33
22:10 MBS DAQ has crashed
Merger terminal is now showing a large number of bad merge events
It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS.
22:42 System wide checks all ok
Statistics - attachment 34
Temperature - Attachment 35
Bias and leakage currents ok - attachment 36
23:33 System wide checks all ok - attachment 37
Temperatures - attachment 38
Bias and leakage currents ok - attachment 39 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|