 Sun Jun 26 08:51:20 2022, OH, NH, Sunday 26 June 08:00-24:00 27x
|
09:51 Taken over from Tom following the night shift
Experiment is still running smoothly
Compression of the files is complete up to the start of R5.
Have started compression of files in R5 which should run up to R5_499
Currently on R5_528
Statistics ok - attachment 1
Temperatures ok - attachment 2
Bias and leakage currents ok - attachment 3
System wide checks:
Base Current Difference
aida07 fault 0xc53d : 0xc594 : 87
aida08 fault 0xf1be : 0xf271 : 179
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3b : 17
FPGA Timestamp error counter test result: Passed 7, Failed 1
Current merger rate is 2E6-4E6 events per second
Current tapeserver rate is 5800 kB/s
Free HDD space is 1.1 TB
At current rates will last 54 hours which should see out the experiment
Experiment currently scheduled to finish at 6am on Tuesday morning (May get until 8am)
Analysis of R5_528 - attachment 4
Deadtime of AIDA02 currently around 15%
11:56 Statistics ok - attachment 5
Temps ok - attachment 6
Bias and leakage currents ok - attachment 7
System wide checks:
Clocks all ok
Base Current Difference
aida07 fault 0xc53d : 0xc59c : 95
aida08 fault 0xf1be : 0xf27a : 188
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3b : 17
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of file R5_458 - attachment 8
Deadtime in AIDA02 only 11.5% in this file
TapeServer rate still 5.5 MB/s
14:36 Has been no beam for the last while or so.
With no beam AIDA02 has 0.25% deadtime so the deadtime is almost entirely due to the spill on time
Stats- attachment 9
Temp - attachment 10
Bias and leakage currents ok - attachment 11
System wide checks:
Clock ok
Base Current Difference
aida07 fault 0xc53d : 0xc59d : 96
aida08 fault 0xf1be : 0xf27c : 190
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3c : 18
FPGA Timestamp error counter test result: Passed 7, Failed 1
13:30 Beam taken to change an ion source
14:57 Beam back
16:21 Statistics ok - attachment 12
Temperatures ok - attachment 13
Bias and leakage currents ok - attachment 14
System wide checks - ASIC clocks ok
Base Current Difference
aida07 fault 0xc53d : 0xc5a3 : 102
aida08 fault 0xf1be : 0xf285 : 199
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3e : 20
FPGA Timestamp error counter test result: Passed 7, Failed 1
Currently on file R5_592
Analysis of R5_591 - Attachment 15
Deadtime of AIDA02 sitting at 17%
[NH taking over for OH so he can get home]
18:53 - FRS DAQ problems mean we weren't taking DESPEC data for the past hour or so... now all back
Statitics ok - attachement 16
Temps ok - attachement 17
Bias & leakage ok - attachement 18
System wide checks -
Clocks OK
ADC Calibration N/A
WR Decoder -
Base Current Difference
aida07 fault 0xc53d : 0xc5ac : 111
aida08 fault 0xf1be : 0xf28c : 206
White Rabbit error counter test result: Passed 6, Failed 2
FPGA -
Base Current Difference
aida07 fault 0x2a : 0x3f : 21
PLL OK
Currently on file R5_621
Analysis of R5_620 - Attachement 19
aida02 deadtime only 5%, all others negligible
20:20 Stats ok - attachment 20
Temps ok - attachment 21
Bias and leakage currents ok - attachment 22
System wide checks:
Clock ok
Base Current Difference
aida07 fault 0xc53d : 0xc5ad : 112
aida08 fault 0xf1be : 0xf293 : 213
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x3f : 21
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R6_637 - attachment 23
22:13 Statistics ok - attachment 24
Temperatures ok - attachment 25
Bias and leakage currents ok - attachment 26
ASIC clock check ok
Base Current Difference
aida07 fault 0xc53d : 0xc5ba : 125
aida08 fault 0xf1be : 0xf2a2 : 228
White Rabbit error counter test result: Passed 6, Failed 2
Base Current Difference
aida07 fault 0x2a : 0x40 : 22
FPGA Timestamp error counter test result: Passed 7, Failed 1
Analysis of R5_658 - attachment 27 |
|
Tue Jun 28 10:11:35 2022, OH, NH, MIDAS Data Aq V10
|
11:11 Rebooted FEEs and changed aidacommon in /MIDAS/linux-ppc_4xx/startup to point to the new V10 DataAq that Patrick produced
When using V9 the Merger statistics reported WR items at twice the rate of ADC data items.
i.e for ever data item we were sending and info code 4 and info code 5 item sending 192 bits of data vs 64 for just the data word
This was causing significant deadtime when FEEs were running in the range of around 200kHz. These WR items were not reported by the MIDAS Acquisition server but were in the Merger statistics
Patrick has produced V10 which removes these.
When running V10 we can confirm in the Merger statistics that this rate is no longer determined by the ADC data rate and instead controlled via Sync Rollover Target in GSI WhiteRabbit Control.
WR items for 0xE - attachment 1
WR items for 0x7 - attachment 2
However we see in the NewMerger terminal the message shown in attachment 3 frequently.
Also we note that the merger time error counter is also going up.
Our thoughts for this are we have a rollover issue (Is the merger expecting the rollover of the LSB to be one value when the MSB is updated but MIDAS is happening on another?)
Are we having dead time issues which is causing time warps?
Does each buffer from the MIDAS Data Acq start with a full WR timestamp?
aidacommon has been changed back to point to V9 to not cause issues when we run the DAQ and forget we changed it to be this way? |
|
Fri Apr 16 07:19:00 2021, OH, MA, Friday 16th April 08:00-16:00 14x
|
08:19 System wide checks all ok
Baselines reset on WR and FPGA checks for the start of the day
Statistics ok - attachment 1
Temperatures ok - attachment 2
Bias and leakage currents ok - attachment 3
09:03 During the reset last night I repaired the OptionsDB for aida02 from the Options.Pristine directory.
No corruptions have been noted since
10:10 System check, clcock and ADC ok
Base Current Difference
aida07 fault 0xfb3a : 0xfb3d : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA and memory check ok
Statistics ok - attachment 4
Temperatures ok - attachment 5
Bias and leakage currents ok - attachment 6
10:39 Fission fragments were punching through AIDA
This was even with the maxmimum degraders in place
They are now placing a thick plate in front of AIDA to block the fragments while they beam tune.
11:38 They have found the issue with the degrader and have corrected it.
We now see next to no implants in AIDA as expect.
Histograms reset at around 11:35
Layout 2 shows very few implants in 5 minutes - attachment 7
12:08 The high implantation rates of ions into AIDA is clearly seen in the large transients on the leakage currents - attachment 8
This was a large amount of dose going into the detectors which we should try to avoid
13:25 system check, clock and ADC ok
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Statistics ok - attachment 9
Temperatures ok - attachment 10
Bias and leakage currents ok - attachment 11
FPGA and memory check ok
16:10 system check, clock and ADC ok
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb3f : 5
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Statistics ok - attachment 12
Temperatures ok - attachment 13
Bias and leakage currents ok - attachment 14
FPGA and memory check ok |
|
Sat Mar 6 06:52:06 2021, OH, LS, Saturday 6th March 30x
|
07:50 Pulser settings - attachment 1
07:52 System wide checks all ok
Statistics ok - attachment 2
Bias ok - attachment 3
09:30 The intensity of fragments is being increased
They are trying to do a factor of 10 increase first which should take us to around 100Hz
10:00 System wide checks all ok
Statistics - attachment 4
Layout 1 showing implants - attachment 5
Fee temperatures ok - attachment 6
12.00 System wide checks all okay, no failures
Statistics (screenshot7)
Spectrum rate (screenshot8)
FEE temps were normal except AIDA04 had "no response" (screenshot9), reloaded several minutes later seems to read normal
(screenshot10)
Leakage currents okay, increase from previous values written to sheets (screenshot12)
Rechecked FEETemps this time AIDA03 had "no response" (screenshot11), seems this occurs when the reload takes longer
than
usual, another reload shows all FEE temps as normal.
Merger and tape server okay
14.00 System wide checks all okay, no failures
Statistics (screenshot13)
Spectrum rate (screenshot14)
FEE temps normal, no issue with reload this time (screenshot15)
Leakage currents okay still increasing, written to sheets (screenshot16)
Merger okay ~4.5M items/sec
Tape server ~4.5Mb/sec
14.50 Beam has been shifted to centre beam more onto AIDA (corresponds to run S452f011)
16.00 System wide checks all okay, no failures
Statistics (screenshot17)
Spectrum rate (screenshot18)
FEE temps normal except AIDA08 "no response" (screenshot19), reload returns to normal (screenshot20)
Leakage currents added to sheet (screenshot21)
Merger okay ~4.5M items/sec
Tape server ~4.5Mb/sec
17.17 no beam, preparing for the 190Ta setting
17:44 OH Takes over
System wide checks all ok
Statistics - attachment 22
Temp ok - Attachment 23
Bias and leakage currents ok - Attatchment 24
While MBS was not currently writing data the merger -> tapeserver link was briefly toggle
The Directory for the experiment was then setup S452
The merger -> Tapeserver link was then re-enabled and data forwarding to MBS has continued.
ASIC Check all ok
18:10 Started writing data to file
File directory S452
Run number R1_0
Seems to have skipped a few files First full file R1_10
Currently writing to /media/SecondDrive/TapeData
Current free space 4211084180 kB
Current write speed 46163 kB/s
Time remaining until full 91222 seconds
25 hours remaining space
18:34 DESPEC starting new MBS file
AIDA Currently on file R1_41
19:00 DESPEC Closing file
AIDA Currently on R1_75
19:05 Comparing last years statistics to this years
FEE Last year This year Factor increase
1 60960 183111 3.00
2 183494 201256 1.09
3 91506 196647 2.15
4 73914 208290 2.81
5 31979 63571 1.99
6 54084 144638 2.67
7 35621 137605 3.86
8 38626 95858 2.48
9 264436 225790 0.85
10 209709 198896 0.94
11 88443 133899 1.51
12 183493 204996 1.12
Sum 1316265 1994557 1.51
19:22 DESPEC opens a new file (They forgot to mention they were closing the previous file)
AIDA on R1_104
Looking int to high write rate to see if we can do anything to reduce it
It isn't the correlation scaler as that is coming in at a rate of 16kHz which amounts to around 1.5MB/s - attachment 24
19:46 Much of the rate is coming from a small number of channels - attachment 25
19:17 WR Timestamp error.
Mentioned to Nic and he said that sometimes small glitches like this are observed
Base Current Difference
aida01 fault 0x9e8b : 0x9e8d : 2
aida02 fault 0x9a8f : 0x9a91 : 2
aida03 fault 0x29e9 : 0x29eb : 2
aida04 fault 0x7f66 : 0x7f68 : 2
aida05 fault 0x1f4f : 0x1f51 : 2
aida06 fault 0x8faa : 0x8fac : 2
aida07 fault 0x53f0 : 0x53f2 : 2
aida08 fault 0x7066 : 0x7068 : 2
White Rabbit error counter test result: Passed 4, Failed 8
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
All other checks passed
20:52 DESPEC Stopping file
AIDA on R1_221
20:53 DESPEC Stopped file because UCESB crashed
AIDA on file R1_237
21:20 All system wide checks ok
Statistics DISC info- Attachment 27
FEE Temp ok - Attachment 28
Bias and leakage current ok - Attachment 29
Statistics good events - Attachment 30
20:51 Started compressing data with command nice -n 10 gzip -v R1_*
Have discussed with Helena and Nic about the potential issues we could run into with the high data rate
I am not sure if MBS can keep up with this rate
We are still writing to file though.
23:00 Correlations have been lost with the FRS DAQ and the rest of the DESPEC analysis
23:10 UCESB restarted and correlations have been regained. Verified with the implant-FRS time difference peak at 13us |
|
Wed Mar 10 07:16:02 2021, OH, LS, Wednesday 10th March 08:00-24:00 39x
|
08:16 System wide checks all ok
Statistics - attachment 1
Temperatures - attachment 2
bias - attachment 3
Merger running 4.5e6 events per second
Tape data writing at 42MB per second
Currently no beam
09:45 While there is no beam we will perform a longer test of the new merger.
Run R20 stopped and merger changed to neew version
Started R21.
The now usual behaviour of additional files at start of merge observed.
N.B. That there was no toggling of the merger no storage or tapeserver no storage. Both were running before the DAQ was
set to going
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_0
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_1
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_2
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_3
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_4
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_5
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_6
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_7
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_8
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_9
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_10
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_11
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_12
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_13
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_14
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_15
-rw-rw-r--. 1 npg npg 64K Mar 10 09:50 R21_16
-rw-rw-r--. 1 npg npg 407M Mar 10 09:50 R21_17
-rw-rw-r--. 1 npg npg 1.6G Mar 10 09:52 R21_18
TapeData rate of 13280 kB/sec
No errors observed in ucesb
Correlations seen between time machine in AIDA and Ge in online
Pulser rate in the online makes sense
With the current data rate remaining HDD space on current drive will last around 89 hours.
09.20 analysis of file R21_27
ignore rates/elapsed idle time - timestamp incomplete until first info code 4 & 5 data
10:24 Message while performing system wide checks:
Get returned with an error
error: SOAP http transport timed out after 10000 ms
NONE
error: SOAP http transport timed out after 10000 ms
while executing
"$transport $procVarName $url $req"
(procedure "::SOAP::invoke" line 18)
invoked from within
"::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get 10"
("eval" body line 1)
invoked from within
"eval ::SOAP::invoke ::SOAP::_XAIDAAccessClient__Get $args"
(procedure "XAIDAAccessClient__Get" line 1)
invoked from within
"XAIDAAccessClient__Get $Addr"
aida02 restarted itself during the reset process
Looking at the log messages on aida02 cannot see any reason for the cause of the restart.
System wide checks following the restart all ok.
When restarting the MBS relay errors observed until a timestamp was observed:
Warning: MBSTimeF = 0; 0x0000000000000000 0x00000000 0x00000000 0x00000000
10:55 Statistics - attachment 5
Temperature - attachment 6
Bias - attachment 7
11:03 Time machine correlation spectra with new merger
AIDA - FATIMA - Attachment 8
AIDA - Ge - Attachment 9
11:30 An implant rate observed in DSSD2. With no beam. A check of the ASIC control restored the rate to 0. Did not check the
layout to determine which FEE/ASIC caused the events before checking ASIC control
12:12 System wide checks all ok *except adc calibration which is same as before
Statistics - attachment 10
Temperature - attachment 11
Bias and leakage currents ok - attachment 12
A note on the statistics. aida11 has doubled in rate today and aida07 has gone down somewhat.
After talking with the DESPEC locals. Helena and Juergen entere at S4 at 9:40 German time.
They stood on the platform but stayed away from the snout, they added two channels to the scope and adjusted the ribbon
cable from the VME scalers.
Looking at the leakage currents on Grafana at 9:50 German time a fluctuation can be observed in the leakage currents of
DSSD3.
12:50 We have waveforms for some FEES
Layout 7 attachment -14
Layout 8 - attachment 15
14.00 (LS)
Still no beam
System wide checks okay except same as before:
**FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module**
Statistics (attachment16)
Checked rate in aida07 and aida11 following Oscars previous comment, last four statistics attachments(1,5,10,16):
aida07 - 124772, 81773, 95526, 101493 - seems to be increasing back up
aida11 - 72901, 95942, 176139, 93799 - more than doubled but large drop in rate back below 100k will keep an eye on
FEE Temps (attachment17)
Leakage currents written to sheets(attachment18)
Merger ~ 45M items/s
TapeServer~ 14MB/s
14.10 While performing checks informed that some beam is back, and they have started a run file (no. S452f113), AIDA on file
R22_98, rate spectra attached (attachment19)
14.10 During the meeting errors appeared in ucesb, checked these timestamps with the corresponding AIDA files and saw
no timewarps so we are not losing anything.
Restarted MBS relay, errors have not reappeared so far
16.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
Statistics (attachment20)
Rate spectra (attachment21)
FEE Temps (attachment22)
Leakage currents written to sheets(attachment23)
Merger ~ 4.3M items/s
TapeServer~ 14MB/s
Current file R22_148
16.26 Increase rate in AIDA as target slits have been opened wider (attachment24), corresponding to runs starting from
S452f115
Around R22_150
18.00 Analysis of R22_122 (before slit adjustments think +-3mm) and R22_178 (after slit adjustment to +-5mm)
Rates of R22_122 (attachment25):
First DSSSD (fee0-fee3) ~66/s
Second DSSSD (fee4-fee7) ~49/s
Third DSSSD (fee8-fee11) ~22/s
Rates of R22_178 (attachment26):
First DSSSD (fee0-fee3) ~122/s
Second DSSSD (fee4-fee7) ~94/s
Third DSSSD (fee8-fee11) ~49/s
18.10 System wide checks all okay except aida07 and aida10 fail calibration (same as previous checks)
Statistics (attachment27)
Rate spectra (attachment28)
FEE Temps (attachment29)
Leakage currents written to sheets(attachment30), look to be on the way down
Merger ~ 4.5M items/s
TapeServer~ 15MB/s
18.24 Several timestamp errors again which also happened earlier (14.10) restarted MBS relay like earlier
19:34 MBS DAQ Crashed
20:00 DAQ is still down. They have Sultan working on it.
20:21 DAQ is back but there are issues with land04 (The raid array data is written to. It was taken out during the lustre
reboot)
We are borrowing a HDD from the SHIP group
20:30 System wide checks all ok
Statistics - attachment 31
Temperature - attachment 32
Bias - attachment 33
22:10 MBS DAQ has crashed
Merger terminal is now showing a large number of bad merge events
It's taken a while but we have managed to get the DAQ back. We had to reset the AIDA MBS.
22:42 System wide checks all ok
Statistics - attachment 34
Temperature - Attachment 35
Bias and leakage currents ok - attachment 36
23:33 System wide checks all ok - attachment 37
Temperatures - attachment 38
Bias and leakage currents ok - attachment 39 |
|
Thu Mar 11 07:09:25 2021, OH, LS, Thursday 11th of March 29x
|
08:00 System wide checks all ok *exctept aida07 and 10 on ADC calibration
N.B. wrong date given to the screenshots by accident (It's early)
Statistics - attachment 1
FEE temperatures (AIDA10 unavailable) - attachment 2
FEE temperature - attachment 3
Bias and leakage currents ok - attachment 4
08:34 The FRS stopped receiving ions. AIDA crashed simultaneously with it.
Have now recovered AIDA daq - Looking at the messages log AIDA04 restarted itself.
Restarted but no stats in aida04
It did not recover from a reset
Telnet in and did a reboot command
09:10 Finally recovered issue was it was not linking to merger
There is an options issue with aida12. The ASIC settings had become undefined and there are no histograms
Manually set the ASIC settings for aida12. Rates look as expected after
Will try to reset once beam is back
10:12 Problem with the DAQ so stopping AIDA while they work will switch MBS relay out to new version
Analysis of R30_21 looks fine
10:38 System wide checks ok
Statistics - attachment 5
Temperature - attachment 6
Bias and leakage - attachment 7
12.10 (LS)
System wide checks all okay, no fails
Statistics (attachment8)
Rate spectra (attachment9)
FEE temps (attament10)
Leakage currents written to sheet (attachment11)
Merger ~45M items/s
Tapeserver ~14MB/s
13.45 no beam (around file R30_98)
14.00 System wide checks all okay, no fails
Statistics (attachment12)
FEE temps (attament13)
Leakage currents written to sheet (attachment14)
Merger ~44M items/s
Tapeserver ~14MB/s
Beam still down
14.11 beam back aida file R30_112
14.13 beam back down, back at 14.16
14.20 no beam
14.23 notice some timestamp error in ucesb 37 minutes ago (about R30_102) no errors appeared on the MBS relay terminal,
double by analysing R30_101,102,103 no timewarps
beam is back but low intensity at the moment
14.46 beam looks to be back at a higher intensity now R30_127 (attachment15)
16.00 System wide checks all okay, no fails
Statistics (attachment16)
Rate spectra (attachment17)
FEE temps (attament18)
Leakage currents written to sheet (attachment19)
Merger ~45M items/s
Tapeserver ~14MB/s
18.00 System wide checks all okay, no fails
Statistics (attachment20)
Rate spectra (attachment21)
FEE temps (attament22)
Leakage currents written to sheet (attachment23)
Merger ~45M items/s
Tapeserver ~15MB/s
bunch of timewarp errors in ucesb, no errors in MBS relay terminal though
now see error in MBS relay!
"Warning: At least one MIDAS block missed in relay" Warning was set up by NH and OH to detect if the LSB of the first ADC data item in a block is less than the LAST LSB of the previous data block. It then iterates the info code 4 and if needed 5 stored before resetting the LSB stored.
It triggered 63 times.
I have checked all the files in the vicinity and there are no timewarps in the MIDAS data visible to TD's analyser
19:10 MBS DAQ stopped to allow FRS tests
19:38 MBS DAQ is back
20:15 Aida07 fails FPGA check
Base Current Difference
aida07 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
All other system wide checks ok
Statistics - attachment 24 (Again ignore the year in the filename)
Temperatures - attachment 25
Bias and leakage currents - 26
21:36 They are resetting the time sorter due to FATIMA issues
22:38 System wide checks error on WR in addition to the previous FPGA error
Base Current Difference
aida05 fault 0x5cc9 : 0x5cca : 1
aida06 fault 0xde76 : 0xde77 : 1
aida07 fault 0xa566 : 0xa567 : 1
aida08 fault 0x6ab0 : 0x6ab1 : 1
White Rabbit error counter test result: Passed 8, Failed 4
Statistics - attachment 27
Temperature - attachment 28
Bias and leakage current - attachment 29
23:02 We still see timewarp errors in the ucesb. With our changes to the relay we can confirm that the WR headers in the MBS relay are time ordered. The issue must be downstream in the MBS chain. |
|
Thu Mar 12 08:04:24 2020, OH, CA, TD, Thursday 12th March 08:00-16:00 14x
|
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
09:00 Bias and leakage currents ok - attachment 1
Statistics ok - attachment 2
FEE temperatures ok - attachment 3
System wide checks all ok
09:08 No beam, not sure exactly how long it has been down.
FRS team are working on it
09:28 still no beam. issues with SIS.
DAQ set to NoStorage mode
most recent file R7_522
10.27 beam is back
writing to file R9_0
10.43 attachment 4 - hit rate spectra
11:09 all system wide checks ok
FEE64 temps ok - attachment 5
good event statistics ok - attachment 6
detector bias and leakage currents - attachment 7
11.18 merger ok, ~ 2.7 million data items per second
TapeServer ok, ~ 25 Mb/s data rate
Data forwarding to MBS ok
11.20 writing to file R9_39
11.22 beam off
11.24 beam back online (R9_42)
11.41 FRS has reverted to an older setting
13:04 hit rate spectra - attachment 8
13:19 all system wide checks ok
13:30 FEE64 temps ok - attachment 9
good event statistics ok - attachment 10
detector bias and leakage currents ok - attachment 11
15:31 good event statistics ok - attachment 13
detector bias and leakage currents ok - attachment 14
FEE64 temps ok - attachment 15
all system wide checks ok
beam seems to have stopped during 2pm meeting
15.35 TapeServer ok, data rate ~ 23 Mb/s
Merger ok, rate of 2.5 - 3 million data items per second
data forwarding to MBS ok
15.38 AIDA writing to file R9_233
16:02 beam back - AIDA writing to file R9_250
|
 Fri May 13 06:56:14 2022, OH, BA, MA, Low Rates
|
| Quote: |
|
The rate is very low in all aida, not sure .
|
This was because the statistics page had been changed to transfer buffers. The rates were ok |
|
Thu May 12 00:37:52 2022, OH, AM, Thursday 12 May 13x
|
01:37 Over the past hour the degrader settings have been checked by impinging 209Pb onto the detectors.
That is now complete.
Will find optimum thresholds for n+n side and then leave DAQ running with n+n at optimum and p+n at 0xc
R16 slow comp 0x19 (250keV) 2 and 4 low deadtime 6 and 8 high deadtime. - attachment1
Will lower 2 and 4 to 200 and raise 6 and 8 to 300
R17 2 good, 4 shakey 6 and 8 poor - attachment 2
R18 2 and 4 at 220 and 6 and 7 at 350
2 and 8 low deadtime, 4 and 6 could be better. - attachment 3
Will run with p+n at 0xc 2+4 at 0x16 and 6 and 8 at 0x23
Statistics at this rate - attachment 4
02:18 Statistics - attachment 7
Temperatures - attachment 6
Bias & leakage current - attachment 5
Clock Status - OK
White Rabbit - OK
The other 3 checks currently aren't working
04:11 Statistics - attachment 10
Temperatures - attachment 9
Bias & leakage current - attachment 8
Clock Status - OK
White Rabbit - OK
06:05 Statistics - attachment 13
Temperatures - attachment 12
Bias & leakage current - attachment 11
Clock Status - OK
White Rabbit - OK |
|
Tue May 4 12:11:39 2021, OH TD, Bias test of triple
|
13:11 System wide checks all ok except *aida02 fails ADC calibration*
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Tue May 04 10:57:38 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Thu Apr 29 14:43:46 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Thu Apr 29 14:43:50 CEST 2021
FEE : aida04 => Options file size is 1014 Last changed Thu Apr 29 14:43:53 CEST 2021
FEE : aida05 => Options file size is 1014 Last changed Thu Apr 29 14:43:55 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Thu Apr 29 14:43:59 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Thu Apr 29 14:44:02 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Thu Apr 29 14:44:05 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Thu Apr 29 14:44:08 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida12 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida13 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida14 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida15 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
FEE : aida16 => Options file size is 1014 Last changed Thu Apr 29 14:44:57 CEST 2021
ASIC clocks have been synchronised and ADC re-calibrated
Rate spectra - attachment 1
Layout7 - attachment 2
Layout8 - attachment 3
Detectors at 130V
Statistics - attachment 4
V-I Curve - attachment 5
Voltage Ch0 Ch1
10 1.345 1.49
20 2.01 2.285
30 2.405 2.8
40 2.635 3.13
50 2.82 3.37
60 2.95 3.57
70 3.08 3.725
80 3.185 3.865
90 3.26 3.98
100 3.33 4.105
110 3.42 4.22
120 3.525 4.35
130 3.64 4.46
140 3.79 4.37
150 3.98 4.37
150+10min 3.9 4.485 |
|
Sun Mar 14 08:16:00 2021, OH PJCS, Sunday 14th March 2021
|
> 08:07 Realised the correlation scalers hadn't been enabled again. Have enabled them now. R56_151
>
> Looking through the merger messages from the nightshift we see bad timestamps in FEE 11. This FEE does not have any scalers going into it.
> It is also the least noisy FEE in the back DSSD
>
> MERGE Data Link (20510): bad timestamp 10 3 0xc2b77eff 0x037a3b92 0x000017f7f37a3b92 0x166c17f7f37a3b92 0x166c17f7f3877262
> MERGE Data Link (20510): bad timestamp 10 3 0xc2827f3a 0x037aa122 0x000017f7f37aa122 0x166c17f7f37aa122 0x166c17f7f3877262
>
> 09:01 System wide checks
>
> Base Current Difference
> aida07 fault 0x8c74 : 0x8c77 : 3
> White Rabbit error counter test result: Passed 11, Failed 1
>
> Understand the status reports as follows:-
> Status bit 3 : White Rabbit decoder detected an error in the received data
> Status bit 2 : Firmware registered WR error, no reload of Timestamp
> Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
>
> Base Current Difference
> aida07 fault 0x2 : 0x5 : 3
> FPGA Timestamp error counter test result: Passed 11, Failed 1
> If any of these counts are reported as in error
> The ASIC readout system has detected a timeslip.
> That is the timestamp read from the time FIFO is not younger than the last
>
> Statistics - attachment 1
> Temperature - attachment 2
> Bias and leakage current - attachment 3
It might be worth considering decreasing the time between the flushing of slower FEEs. Is the Merger waiting for the slower data and causing ‘deadtime’ in the faster ones. Perhaps consider this as a
balancing across the system? |
|
Tue Jun 21 10:19:05 2022, OH ML, Tuesday 21st June 08:00-00:00 12x
|
11:19 Stats ok 0xa on all FEE - attachment 1
Current pulser settings - attachment 2
FEE Temperatures ok - attachment 3
Noticed both AIDA03 and AIDA07 had ASICs with missing channels. Did a check ASIC control on them to recover
Layout 1 after recovering channels - attachment 4
New stats (Note increased rate on 3 and 7) attachment 5
Bias and leakage currents ok - attachment 6
S4 now in closed access. Trying to find out if the beam dump has been put in front of AIDA. Will make sure this is checked before beam to S4.
14:05 Have had it confirmed that the flange beam dump is in position in S4
15:43 Stats ok -attachment 7
Temp ok - attachment 8
Bias and Leakage current ok - attachment 9
17:45 Currently the time machine trigger(for correlation) is connected to the DTAS or.
As the beam is tuning they are triggering at 150k. This is 450k worth of scalers in AIDA
During the experiment this will be changed to SC1 trigger to solve this.
19:49: Stats ok - attachment 10
temperatures ok - attachment 11
Bias - Leakage current ok - attachment 12 |
|
Thu Sep 20 08:57:48 2018, OH, Thursday 20th September
|
09:57 Alpha run stoppped R2_0 -> R2_2
Thresholds dropped from 0x64 to 0xa
Pulser turned on again
System wide checks ok
FEE Temp ok - attachment 1
Bias ok, leakage at 1.485uA at 160V
Stats in similar place to yesterday - attachment 2
Low Energy pulser walkthrough
1.0V to 0.4V in 0.1V steps
2 minutes per step
Pulser at 50Hz
10:29 Run started R3_0
10:45 Run stopped R3_2
1.8.L spectra for low energy pulser walkthrough - attachment 3
11:00
High Energy pulser walkthrough
1.8V to 0.V in 0.2V
2 minutes per step
Pulser at 50Hz
11:06 Run started R4_0
10:18 Run stopped R4_2
1.8.L spectra for high energy pulser walkthrough - attachment 4 |
|
Tue Mar 10 10:16:56 2020, OH, AIDA Implant Range Scan
|
A range scan was performed using AIDA.
See the GSI elog: https://elog.gsi.de/despec/S480/19 |
|
Wed Mar 11 23:44:27 2020, OH, Thursday 12th March 00:00-08:000 10x
|
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
00:44 Implantation range in AIDA - attachment 1
01:14 At current rate will run out of space in 35 hours ~ 18:00 hours on Friday 13th March.
During the day if possible it might be wise to try and obtain a USB3 HDD mount to mount one of the spare drives in
the AIDA boxes and begin moving files onto that
01:35 System wide checks all ok
Bias and leakage current ok - attachment 2
Stats all ok - attachment 3
FEE temperatures ok - attachment 4
TapeServer rate around 25-29MB per second
Merger rate around 2.6-3.2 million items per second
04:11 System wide checks all ok
Bias and leakage currents ok - attachment 5
Stats all ok - attachment 6
FEE temp ok - attachment 7
06:38 System wide checks all ok
Bias and leakage currents ok - attachment 8
Stats all ok - attachment 9
FEE temp ok - attachment 10 |
|
Thu Mar 4 10:13:48 2021, OH, AnyDesk connection instuction
|
|
|
Thu Mar 4 11:16:13 2021, OH, Further cases of possible database issues
|
During testing for the S452 experiment yesterday we were running through the steps of a restart.
The FEEs were not powercycled at this stage but the MIDAS server was restarted.
Initial setup went smoothly and no issues were encountered during setup.
NewMerger and TapeServer were both setup and set to going.
Upon ''Going'' the run control the data transfer for aida09 dropped out.
Going to the NewMerger it could be seen that data was making it through and data items were being merged.
The tapeserver however was seeing no data and neither was the MBS dataspy.
During this time the FEEs became unresponsive likely as their buffers filled. Trying to reset gave the attached errors.
Checking the folder manually it could be seen that the Options file was there.
At this point a powercycle was pewrformed but upong 'Going' the DAQ the same issue was again encountered.
This time we were able to recover control access to the FEEs by running multiple NewMerger sessions which helped clear the buffers.
During our search for the problem we looked at the Options from within MIDAS and could not see anything out of the ordinary.
We also tried restoring the Options from within MIDAS.
We again reached the same issue with the tapeserver.
At this point we restored the Options folder from by copying manually within terminal.
Following this we were able to start the DAQ normally. |
|
Fri Mar 5 22:53:05 2021, OH, Thursday 5th of March
|
There has been no beam for much of the day. It was lost at 13:00 and not regained until 22:00 German time.
At some point during the day the White Rabbit timestamp cable was unplugged from AIDA. This caused it lose correlation with the rest of the system.
At first a reset and setup was performed to try and regain correlation but that did not prove successful.
A full powercycle was then performed at ~22:30 German time which upon restarting the DAQ solved the issue.
The plan is to overnight run some fragments, for the purposes of observing the isomers. This data is not essential and as such we will not be writing to disk.
Beam plug is in front of AIDA. This is causing a huge spike in the Ge. |
|
Sun Mar 7 23:13:50 2021, OH, Monday 8th March 13x
|
00:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:14 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
01:20 Can see now that we are writing to a separate drive the compression is going much faster - attachment 4
02:15 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 5
FEE Temperatures - attachment 6
Bias and leakage currents ok - attachment 7
03:43 Beam has been lost. Not and FRS fault. Several powersupplies have tripped
04:00 Noticed more bad merge events
System wide checks has also noted an FPGA error.
All other system wide checks same results as previous
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A3F29 BA0DF4D2 , WR/10=> 23DD31DC5CE3215, Readout Time => 23DD31DC6F34000
aida02 : White Rabbit=> 166A3F29 CC49ECD7 , WR/10=> 23DD31DC7A0FE15, Readout Time => 23DD31DC8428000
aida03 : White Rabbit=> 166A3F29 DF55B7E8 , WR/10=> 23DD31DC9889264, Readout Time => 23DD31DCA85C000
aida04 : White Rabbit=> 166A3F29 EECF82B7 , WR/10=> 23DD31DCB14C045, Readout Time => 23DD31DCC2B8000
aida05 : White Rabbit=> 166A3F29 FED2EFD2 , WR/10=> 23DD31DCCAEB195, Readout Time => 23DD31DCDC14000
aida06 : White Rabbit=> 166A3F2A 0EDC023D , WR/10=> 23DD31DCE49336C, Readout Time => 23DD31DCF204000
aida07 : White Rabbit=> 166A3F2A 1E8170D3 , WR/10=> 23DD31DCFD9BE7B, Readout Time => 23DD31DD10A0000
aida08 : White Rabbit=> 166A3F2A 30448F76 , WR/10=> 23DD31DD1A074BF, Readout Time => 23DD31DD2A34000
aida09 : White Rabbit=> 166A3F2A 3FE36E61 , WR/10=> 23DD31DD33057D6, Readout Time => 23DD31DD3B94000
aida10 : White Rabbit=> 166A3F2A 4E83A1F7 , WR/10=> 23DD31DD4A6C365, Readout Time => 23DD31DD46A8000
aida11 : White Rabbit=> 166A3F2A 5FD19086 , WR/10=> 23DD31DD661C1A7, Readout Time => 23DD31DD7664000
aida12 : White Rabbit=> 166A3F2A 6FE06F3D , WR/10=> 23DD31DD7FCD7EC, Readout Time => 23DD31DD9684000
04:07 Beam is back AIDA on file R6_815
Statistics - attachment 8
FEE Temperatures - attachment 9
Bias and leakage currents ok - attachment 10
ASIC check ok
Multiple merge data error events observed
06:04 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 11
FEE Temperatures - attachment 12
Bias and leakage currents ok - attachment 13
07:22 Beam noticed dropped but back quickly |
|
Mon Mar 8 22:51:47 2021, OH, Tuesday 9th March 10x
|
18:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
00:12 Noticed this log in UCESB.
Closed connection [140.181.60.97]...
1 clients...
AIDA Timewarp (166a7a92f8421252 before 166a7a92fdad3848)
AIDA timewarp is over, skipped 2728 AIDA event(s)
=> Not emitting timewarped event (before 166a7a92fdac5c02)
Recovered from timewarp but skipped 5 event(s)
at 00:30 this was 7853.9854 seconds ago or 2 hours and 11 minutes ago which would be 22:19 possibly R9_957 ->R9_961
Tracked the timestamp down to R9_954 found no evidence of timewarp in analysis of file not in R9_255 either
Checked all files from R9_254->R9_263 found no timewarps and covers the span of timestamps mentioned in message
01:09 Another timewarp message
AIDA Timewarp (166a83bc5badee1a before 166a83bc6bade974)
=> Not emitting timewarped event (before 166a83bc6226e9ea)
Recovered from timewarp but skipped 6 event(s)
AIDA timewarp is over, skipped 323841 AIDA event
Time relates to Your time zone: Tuesday, 9 March 2021 00:04:21.649 GMT+00:00 this corresponds to R9_1175
Again no timewarps in analysis of file.
02:00 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 4
FEE Temperatures - attachment 5
Bias and leakage currents ok - attachment 6
03:10 Problem with accelerator will be no beam for ~30 minutes
03:56 There was a huge spike of bad merge events. AIDA also stopped forwarding data to the MBS relay while this spike was taking place.
There are now also errors in all FEEs for WR events
Base Current Difference
aida01 fault 0xc45f : 0xc46a : 11
aida02 fault 0xd250 : 0xd25b : 11
aida03 fault 0x27b2 : 0x27bd : 11
aida04 fault 0x714b : 0x7156 : 11
aida05 fault 0x74f5 : 0x7500 : 11
aida06 fault 0x558a : 0x5595 : 11
aida07 fault 0x8d20 : 0x8d2b : 11
aida08 fault 0x8131 : 0x813c : 11
aida09 fault 0xf412 : 0xf41d : 11
aida10 fault 0x82c2 : 0x82cd : 11
aida11 fault 0x2845 : 0x2850 : 11
aida12 fault 0x1bf6 : 0x1c01 : 11
White Rabbit error counter test result: Passed 0, Failed 12
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Also errors in FPGA
Base Current Difference
aida03 fault 0x0 : 0x1 : 1
aida04 fault 0x0 : 0x1 : 1
aida10 fault 0x0 : 0x2 : 2
aida11 fault 0x0 : 0x1 : 1
aida12 fault 0x1 : 0x2 : 1
FPGA Timestamp error counter test result: Passed 7, Failed 5
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A8D22 3A5FDE66 , WR/10=> 23DDAE9D2A32FD7, Readout Time => 23DDAE9D3C9C000
aida02 : White Rabbit=> 166A8D22 4C6C3AEB , WR/10=> 23DDAE9D4713917, Readout Time => 23DDAE9D5B94000
aida03 : White Rabbit=> 166A8D22 5FB33A3C , WR/10=> 23DDAE9D65EB906, Readout Time => 23DDAE9D769C000
aida04 : White Rabbit=> 166A8D22 6FBA29DB , WR/10=> 23DDAE9D7F9042F, Readout Time => 23DDAE9D8D14000
aida05 : White Rabbit=> 166A8D22 7F71248D , WR/10=> 23DDAE9D98B5074, Readout Time => 23DDAE9DA9D8000
aida06 : White Rabbit=> 166A8D22 8F78B071 , WR/10=> 23DDAE9DB25AB3E, Readout Time => 23DDAE9DC280000
aida07 : White Rabbit=> 166A8D22 9F5A64B7 , WR/10=> 23DDAE9DCBC3D45, Readout Time => 23DDAE9DDFF0000
aida08 : White Rabbit=> 166A8D22 B1E91092 , WR/10=> 23DDAE9DE974E75, Readout Time => 23DDAE9DFA08000
aida09 : White Rabbit=> 166A8D22 C150D9ED , WR/10=> 23DDAE9E021AF64, Readout Time => 23DDAE9E0FD8000
aida10 : White Rabbit=> 166A8D22 D02BA583 , WR/10=> 23DDAE9E19DF6F3, Readout Time => 23DDAE9E2D50000
aida11 : White Rabbit=> 166A8D22 E287CFC2 , WR/10=> 23DDAE9E373FB2D, Readout Time => 23DDAE9E4720000
aida12 : White Rabbit=> 166A8D22 F2D4C911 , WR/10=> 23DDAE9E515474E, Readout Time => 23DDAE9E1BCC000
System recovered and started forwarding data to MBS again but then had another spike in bad merge events with the same results
AIDA DAQ Stopped (Still no beam)
The problem is extraction from the SIS (That is the problem with the beam and not the problem with AIDA).
03:23 AIDA recovered from power cycle. This was done as previously it was observed that after white rabbit issues (When the fibre optic was unplugged) a reset alone was not able to resume synchronisation between AIDA and the other systems. AIDA recovered from the power cycle uneventfully.
All system wide checks in AIDA now pass. (AIDA06 no longer shows the error)
Currently running to no storage on the TapeServer will resume with R13 once beam is back. Forwarding to MBS again.
05:27 Accelerator operators say beam should be back. We see no evidence of this in the triggers or AIDA though - attachment 7
The problem was with the accelerator operators system. There was no beam.
06:20 Error in WR system wide checks. Rest all ok
This time there is no bad merge items recorded in the merger terminal.
There are also no timewarp events seen in ucesb which were observed with the issues at 3am.
Base Current Difference
aida01 fault 0xdbe9 : 0xdbea : 1
aida02 fault 0xa93d : 0xa93e : 1
aida03 fault 0x8fc4 : 0x8fc5 : 1
aida04 fault 0x3e56 : 0x3e57 : 1
aida05 fault 0xb04c : 0xb04d : 1
aida06 fault 0x5576 : 0x5577 : 1
aida07 fault 0x99ff : 0x9a00 : 1
aida08 fault 0x8412 : 0x8413 : 1
aida09 fault 0x2457 : 0x2458 : 1
White Rabbit error counter test result: Passed 3, Failed 9
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
The following are timestamp values from each of the FEEs taken in sequence
If time does not increase in a reasonable manner run the system wide checks
aida01 : White Rabbit=> 166A9523 11478D1D , WR/10=> 23DDBB6B4ED8E1C, Readout Time => 23DDBB6B6148000
aida02 : White Rabbit=> 166A9523 236AA5E8 , WR/10=> 23DDBB6B6BDDD64, Readout Time => 23DDBB6B8164000
aida03 : White Rabbit=> 166A9523 375F235D , WR/10=> 23DDBB6B8BCB6BC, Readout Time => 23DDBB6B9C3C000
aida04 : White Rabbit=> 166A9523 47B8038C , WR/10=> 23DDBB6BA5F338E, Readout Time => 23DDBB6BA77C000
aida05 : White Rabbit=> 166A9523 599FC116 , WR/10=> 23DDBB6BC29934F, Readout Time => 23DDBB6BD34C000
aida06 : White Rabbit=> 166A9523 696EB0A7 , WR/10=> 23DDBB6BDBE44DD, Readout Time => 23DDBB6BECB8000
aida07 : White Rabbit=> 166A9523 7982236B , WR/10=> 23DDBB6BF59D057, Readout Time => 23DDBB6C0418000
aida08 : White Rabbit=> 166A9523 87C85008 , WR/10=> 23DDBB6C0C73B34, Readout Time => 23DDBB6C1C7C000
aida09 : White Rabbit=> 166A9523 97020497 , WR/10=> 23DDBB6C24D0075, Readout Time => 23DDBB6C34B4000
aida10 : White Rabbit=> 166A9523 A69B171D , WR/10=> 23DDBB6C3DC4F1C, Readout Time => 23DDBB6C50A0000
aida11 : White Rabbit=> 166A9523 B86C8D7D , WR/10=> 23DDBB6C5A4748C, Readout Time => 23DDBB6C6C88000
aida12 : White Rabbit=> 166A9523 CA1DD1B2 , WR/10=> 23DDBB6C76961C5, Readout Time => 23DDBB6C5C40000
Possibly occured at 05:28 as a ucesb reports a timewarp at the time.
AIDA Timewarp (166a922b75368928 before 166a922b799e6c92)
AIDA timewarp is over, skipped 2728 AIDA event(s)
Cannot check file as was not writing to file.
Statistics - attachment 8
FEE Temperatures - attachment 9
Bias and leakage currents ok - attachment 10
06:40 There are occasional seconds of beam. According to operators it comes and then it goes again
06:45 Beam fairly stable so we are running again.
AIDA starts R13. Once again it has skipped many files when unselecting no storage and started on R13_19
07:00 Beam stopped again
07:02 Beam returned
There are apparently issues with MUSIC chamber one and its resolution
07:46 Beam is still intermitent. There now seems to be problems with the Go4 analysis as it keeps crashing. |
|