 Fri Apr 16 23:13:57 2021, CA, MA, BA, Saturday 16th 00:00 - 08:00 14x
|
00:05 DESPEC tuning beam, changing spill length
00:21 DAQ continues ok
Merger/TapeServer ok
00:25 still no beam in AIDA
00:30 statistics ok, all FEE64 showing rates
no error messages in database check
AIDA scalars on ucesb ok - attachment 1
00:52 beam in AIDA - rates spectra - attachment 2
01:05 statistics ok, all FEE64 showing rates
no error messages in database check
AIDA scalars on ucesb ok - attachment 3
beam is off again
01:30 beam back
01:32 statistics ok, all FEE64 showing rates
no error messages in database check
AIDA scalars on ucesb ok - attachment 4
01:37 beam off - FRS increasing spill length to 5s
01:48 beam back with new spill setting - 5s on spill, 2s off spill
02:07 system wide checks ok except;
WR decoder status
Base Current Difference
aida05 fault 0xc879 : 0xc87b : 2
aida06 fault 0x323c : 0x323e : 2
aida07 fault 0xfb3a : 0xfb42 : 8
aida08 fault 0xd3d6 : 0xd3d8 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FPGA check :
Base Current Difference
aida07 fault 0xf : 0x10 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
02:10 FEE64 Temps ok - attachment 5
02:20 good event statistics ok - attachment 6
02:27 detector bias / leakage currents ok - attachment 7
02:30 statistics ok, all FEE64 showing rates
no error messages in database check
AIDA scalars on ucesb ok
beam off
03:00 stats/ucesb/database checks ok
03:30 stats/ucesb/database checks ok
still no beam in AIDA
05:11 system is checked, beam is back with low intensity
05:13 FEE64 Temps ok - attachment 8
05:13 good event statistics ok - attachment 9
05:13 detector bias / leakage currents ok - attachment 10
05:33 stats/ucesb/database checks ok
06:00 AIDA crashes, lose connection to aida09
06:20 aida08 does not respond on powercycle and reset
06:30 Called TD
06:50 successfully reset AIDA. Writing to file R30
synchronise ASIC clocks:
FEE64 module aida01 => 7
FEE64 module aida02 => 7
FEE64 module aida03 => 7
FEE64 module aida04 => 7
FEE64 module aida05 => 7
FEE64 module aida06 => 7
FEE64 module aida07 => 7
FEE64 module aida08 => 7
FEE64 module aida09 => 7
FEE64 module aida10 => 7
FEE64 module aida11 => 7
FEE64 module aida12 => 7
ASIC Clock Timestamp check test result: Passed 12, Failed 0
Database options file size check:
If any of the above values are different from the rest then check the clocks and the White Rabbit decoder status
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Sat Apr 17 06:14:30 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Wed Apr 14 21:58:54 CEST 2021
System wide checks ok *except*
FEE64 module aida06 failed
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 9, Failed 3
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
re-calibration of these FEE modules unsuccessful in FADC align and control.
07:05 FEE64 Temps ok - attachment 12
good event statistics ok - attachment 13
detector bias / leakage currents ok - attachment 14
07:10 stats/ucesb/database checks ok
07:30 stats/ucesb/database checks ok
|
|
Wed Mar 11 16:29:17 2020, CA, LS, TD, slow comparator threshold comparison
|
17.29 good event statistics for slow comparator threshold at 100 keV (0xa) - attachment 1
17.36 good event statistics for slow comparator threshold at 110 keV (0xb) - attachment 2
17.39 good event statistics for slow comparator threshold at 120 keV (0xc) - attachment 3
17.53 good event statistics for slow comparator threshold at 80 keV (0x8) for DSSD2, and 100keV (0xa) for DSSD 1&3 - attachment 4 |
|
Sun Mar 15 06:45:00 2020, CA, LS, CG, Sunday 15th March 08:00 - 24:00 35x
|
ASIC settings 2019Oct31-13.24.23
slow comparator 0xa
BNC PB-5 pulser
amplitude 1.0V , attenuator x1
frequency 2Hz
decay time 1ms
07:46 all system wide checks ok
good event statistics ok - attachment 1
FEE64 temperatures ok - attachment 2
detector bias/leakage currents ok - attachment 3
08:48 merger ok ~ 3 million data items/s
TapeServer ok ~ 27 MB/s
data forwarding to MBS ok
08:50 AIDA writing to file SecondDrive/TapeData/S480/R12_1524
09:56 all system wide checks ok
good event statistics ok - attachment 4
detector bias/leakage currents ok - attachment 5
FEE64 temperatures ok - attachment 6
12:09 all system wide checks ok
12:13 all system wide checks ok
good event statistics ok - attachment 7
FEE64 temperatures ok - attachment 8
detector bias/leakage currents ok, but ramping up - attachment 9
13:32 merger ok ~ 3 million data items/s
TapeServer ok ~ 23 MB/s
data forwarding to MBS ok
Beam seems to be off
13:33 AIDA writing to file R12_1743
*update* checked DESPEC S480 Elog, issues with UNILAC, beam lost at 13:29
13:55 beam back
13:57 AIDA writing to file R12_1759
14:35 all system wide checks ok
FEE64 temperatures ok - attachment 10
good event statistics ok - attachment 11
detector bias/leakage currents ok, but ramping up - attachment 12
15:18 merger ok ~ 2.7 million data items/s
TapeServer ok ~ 27 MB/s
data forwarding to MBS ok
AIDA writing to file R12_1283
16:00 1.8.H high energy spectra - attachments 13 & 14
hit-rate spectra - attachment 15
16.22 all system wide checks okay
FEE temperatures okay (attachment 16)
leakage currents okay and recorded to spreadsheet (attachment 17)
good event stats okay (attachment 18)
merger running at 3M items/sec
tape service running at 26MB/sec
18.00 began compressing R7 files with command: nice -n 10 gzip -v R7_* &
all system wide checks okay
FEE temperatures okay (attachment 19)
leakage currents okay and recorded to spreadsheet (attachment 20)
good event stats okay (attachment 21)
merger running at 3M items/sec
tape service running at 27MB/sec
20.10 all system wide checks okay
FEE temperatures okay (attachment 22)
leakage currents okay and recorded to spreadsheet, detectors 1 and 2 still rising (attachment 23)
good event stats okay (attachment 24)
merger running at 3M items/sec
tape service running at 25MB/sec
rates histogram (attachment 25)
low energy histograms (attachment 26 and 27)
high energy histograms (attachments 28 and 29)
22.07 all system wide checks okay
FEE temperatures okay (attachment 30)
leakage currents okay and recorded to spreadsheet (attachment 31)
good event stats okay (attachment 32)
merger running at 3M items/sec
tape service running at 25MB/sec
currently on R12_2097
23.56 all system wide checks okay
FEE temperatures okay (attachment 33)
leakage currents okay and recorded to spreadsheet (attachment 34)
good event stats okay (attachment 35)
merger running at 3M items/sec
tape service running at 26MB/sec
finishing up now. night shift group have been informed how to check up on AIDA, there is a sheet on the table with relevant information
|
|
Sun Mar 7 07:23:26 2021, CA, LS, March 7th 14x
|
08:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xa
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
08:30 - all system wide checks ok
- Temperatures ok - attachment 1
- good event statistics ok - attachment 2
- detector bias/leakage currents ok - attachment 3
DAQ continues to run - R1_1148
Merger ok - 4.6M data items/s
TapeServer ok - 43MB/s
08:55 Giovanna Benzoni - confirms contribution of light ions - attachment 4 see AOQ plots
09:19 zeroed all histograms
writing to file R1_1204
FRS on file 72
DESPEC to add AIDA XY hit patterns to online analysis histograms
09:35 beam stop
09:56 beam back
writing to file R1_1249
rate spectra - attachment 5
09:30 AIDA DAQ crashed - lost connection to aida05
power cycle and reset
09:40 AIDA ok now, writing to file R2
09:50 LS takes over
11.10 (German time)
System wide checks produces some fails:
FEE64 module aida06 global clocks failed, 6 (screenshot6)
This error is seen in RIKEN too, tried a RESYNC from master timestamp tab but error still there
FEE64 module aida06 failed
FEE64 module aida07 failed
Calibration test result: Passed 10, Failed 2 (screenshot7)
Tried a recalibration for the two FEEs in FADC align & control tab but error still there
Rest of the checks pass
Statistics (screenshot8)
FEE temps okay (screenshot9)
Leakage currents written to sheets (screenshot10)
Merger ~4.5M data items/s
TapeServer ~ 45MB/s
13.00 System wide checks produces same fails as last time:
FEE64 module aida06 global clocks failed, 6
FEE64 module aida06 failed
FEE64 module aida07 failed
Calibration test result: Passed 10, Failed 2
Rest of the checks are okay
Statistics (screenshot11)
Spectra rate (screenshot12)
FEE temps okay (screenshot13)
Leakage currents written to sheets (screenshot14)
Merger ~4.5M data items/s
TapeServer ~ 45MB/s
There is 1.5TB left on current drive
14.00 beam stopped to look at bPlast, have use the time to change slow comparator threshold
Slow comparators in FEE9-12 raised to 0xd
|
|
Mon Mar 8 07:02:17 2021, CA, LS, Monday 8th March 08:00 - 17:00 24x
|
08:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude 1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
08:20 Merger ~4.5M data items/s
TapeServer ~ 45MB/s
Writing to file R6_1148
09:00 Beam off for optimisation - writing to file R6_1212
Expected back at 09:30
09:30 Beam will be off for a bit longer - an hour or two
09:34 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 4
FEE Temperatures - attachment 5
Bias and leakage currents ok - attachment 6
Merger ~4.5M data items/s
TapeServer ~ 45MB/s
10:10 (NH) Stopped DAQ to try the new merger from VP, new run R7, R8 for these tests
Tape rate was 110 MB/s (!) and MBS relay seemed nothing... also it didn't stop when DAQ was stopped but took 10+ seconds
to finish
Something confusing, will look at data and report to VP
Reverted to original merger and back... 45 MB/s
R9
12.00 (LS)
System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
Statistics (attachment7)
FEE temps (attatchment8)
Rate spectra (attachment9), beam still down
Leakage Currents written to sheets (attachment10)
Merger ~45M items/s
Tapeserver ~43MB/s
14.00 Still no beam but is expected to return soon
System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
Statistics (attachment11)
FEE temps (attatchment12)
Leakage Currents written to sheets (attachment13)
Merger ~45M items/s
Tapeserver ~44MB/s
14.40 Leakage currents still rising slowly compared to previous days, so will keep an eye on them (attachment14)
15.00 Beam is back (attachment15)
Still writing to R9 (roughly around R9_386)
15.28 No beam (attachment16)
15.41 Beam back (attachment17)
Roughly R9 (roughly around R9_439)
16.00 Analysis of R9_469, from FEE0 and FEE2 the hit rate for one side is around 48 HEC data items/second (attachment18)
16.20 System wide checks okay except 'FEE64 module aida06 global clocks failed, 6'
Statistics (attachment19)
Spectra Rate (attachment20)
FEE temps (attatchment21)
Leakage Currents written to sheets (attachment22),
still rising but looking on Grafana 7 day period should start to lower around 17.00
Merger ~45M items/s
Tapeserver ~43MB/s
16.55 Beam down (attachment23), back a couple minutes later
15.51 beam down (attachment24)
|
|
Tue Mar 9 06:58:14 2021, CA, LS, March 9th 08:00-11:00 shift 23x
|
08:00 DAQ continues ok - writing to file R13_120
Merger ok - 4.6M data items/s
TapeServer ok - 43 MB/s
08:03 - all system wide checks ok
- Temperatures ok - attachment 1
- good event statistics ok - attachment 2
- detector bias/leakage currents ok - attachment 3
08:08 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
08:32 CA briefly lost network connection - back now
08:42 DESPEC on file 102
09:24 Rate spectra - attachment 4
09:32 HEC spectra - attachment 5/6
beam off - writing to file R13_243
09:42 merger bad timestamp errors - attachment 7
all system wide checks ok
10:30 NH - Beam was down, I try the new merger from Vic again (R14)
I made a mistake and had to powercycle fees as they got stuck! aida10 seemed to reboot itslef
After fixing my mistakes, all running with new merger and rate 15 MB/s (R14)
It all looked OK even going to MBS/Go4
I reverted back to original hot merger, R15 and look at the data.
Beam is now back and I return AIDA to shifters :)
11.30 System wide checks okay except ADC calibration:
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Reran calibration for both FEEs but still fail
Statistics (attachment8)
Rate spectra (attachment9)
FEE temps (attachment10)
Leakage currents written to sheet (attachment11)
Tapeserver ~33MB/s
Merger ~ 3.4M items/s
AIDA file R15_92
11.55 Beam off for a couple minutes but back now
12.00 New Merge terminal disappeared so stopped daq and restarted, and also restarted MBS back working
Now writing to R16
Merger ~ 4.5M items/s
Tapeserver ~44MB/s
12.41 no beam
13.00 DAQ crashed (aida03 no responding), AIDA recovered from power cycle while beam is down
Beam is back now, forwarding to MBS
Merger ~ 45M items/s
Tapeserver ~44MB/s
Writing to R17
13.20 System wide checks okay except ADC calibration:
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment12)
Rate spectra (attachment13)
FEE temps (attachment14)
Leakage currents written to sheet (attachment15)
Tapeserver ~45MB/s
Merger ~ 4.4M items/s
14.27 beam off
file R17_115
back a couple minutes later
14.40 no beam again back at 14.46
15.40 System wide checks okay except ADC calibration:
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment16)
Rate spectra (attachment17)
FEE temps (attachment18)
Leakage currents written to sheet (attachment19)
Tapeserver ~44MB/s
Merger ~ 4.4M items/s
15.52 beam down R17_221, back at 15.58 R17_229
16.07 Beam down
16:20 Removed S480 files from /media/1e... and /media/SecondDrive
Set merger to no output and stopped tapedata.
Changed symbolic link of /TapeData to /media/1e.....
Restarted tapedata and the merger with outputting to file R20
Local DESPEC was made aware of this before carrying out and stopped the run.
Beam back
17.12 beam down for a couple of minutes R20_90
17.40 System wide checks okay except ADC calibration:
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun
calibration for that module
Statistics (attachment20)
Rate spectra (attachment21)
FEE temps (attachment22)
Leakage currents written to sheet (attachment23)
Tapeserver ~45MB/s
Merger ~ 4.5M items/s
|
|
Thu Dec 5 22:00:11 2019, CA, DK, NH, Dec 5 Night Shift 30x
|
15:30
TD found the merger was running at ~43 Mb/s
The odd-numbered FEE64s all had significant (>200k) disk rates
ALL LEC/MEC fast discriminators and increased slow comparator from 0xa to 0x14
Merger rate dropped to a more nominal value like 2.3 Mb/s
Settings not saved
40Ar beam 300 MeV/u expected, but SIS problem at the moment.
23:00
Leak currents are stable as previously (20-21 Nov) and entered to Google spreadsheet
S4 vault is closed.
Presently AIDA is running without tape storage
System wide checks:
Master clock / aida01 sync reports error as expected;
All white rabbit decoders fail, but they are just off by 1 bit (no problem)
Everything else okay.
Stats look okay, but keep in mind above discriminator threshold -- see attachment 1
Temperatures fine -- see attachment 2
MBS looks okay -- we see a possible hint for the 1 bit difference with an "unexpected data word" -- see attachment 3
Data rate is 1.7 Mb/s, merger okay -- see attachment 4
23:30
Now we wait for the beam
00:05
Attachments 5-11: results of system wide checks after collecting sync errors and WR errors from baseline - all ok
00:25
Histograms zeroed at 00:25.
00:35
1.8.L pulser peak widths
FEE64 width(ch)
1 92
2 131
3 75
4 97
5 89
6 117
7 90
8 87
9 223
10 58
11 112
12 126
no peak observed on aida09
For comparison, pulser peaks measured on 13/11/19:
pulser peak widths:
FEE width (ch)
aida01 - 130
aida02 - 168
aida03 - 83
aida04 - 83
aida05 - 200
aida06 - 148
aida07 - 250
aida08 - 81
aida09 - 134
aida10 - 190
aida11 - 333
aida12 - 149
2:00
40Ar beam tuning is finished. Intensity at S4 is about 3 kHz per spill. 1 spill per / 10 seconds
MBS system cannot be used for online merging tonight, because NeuLAND group left their connection open
We can merge offline using the White Rabbit timestamps
Remove the beam blocking steel flange from in front of AIDA et al.
Writing to file Dec19/R1
03.20 Rate, Stat, 1.8.L, 1.8.H, 1.8.W spectra - attachments 13-20
Observe
light ions in 1.8.L spectra
low energy (<1.5GeV) heavy ions in 1.8.H spectra with most heavy ions stopping in DSSSDs 1 & 2
Stat hit patterns indicate broad (c. 5cm FWHM) centred heavy ion peak
03.24 Detector biases & leakage currents OK - attachment 21
FEE64 temperatures OK - attachment 22
Good event statistics (12 of 12 < 50k) - attachment 23
System wide checks
List All Sync Error Counters - attachment 24
List all Sync Received Counters - attachment 25
Baseline invoked earlier c. 00.05
Collect all SYNC Error Counters for Baseline
Collect All White Rabbit status Error Counters for Baseline
04:07 2.6 mm Al degrader. 300Hz beam. 300 MeV/u 40Ar beam.
04.14 pulser peak seen in aida09 after performing ASIC check-load earlier in shift - width = 219 ch
04.45 1.8.H energy spectra (attachment 26/27)
06:08 attachments 28-30: detector bias/leakage currents ok
temperatures ok
good event statistics ok
all system wide checks ok
08:49 DAQ stop/start. TapeServer writing to No Storage. End of beam.
|
|
Fri Dec 6 20:34:56 2019, CA, DK, NH, Dec 6 Night Shift 15x
|
21.35 AIDA DAQ currently running, TapeService on, not writing to storage
System wide checks ok
Attachments 1-3: temperatures, good event stats, detector bias/leakage currents ok
22.04 waiting for beam. Possibility of fragments, further news to come
22.43 expecting fragments with 300 MeV/u 40Ar primary beam
histograms zeroed
00.31 still waiting for beam
detector bias/leakage currents ok (see google spreadsheet)
FEE64 temps and good event statistics ok
02:02 still awaiting beam - issues with interlock
03:17 beam tuning - centering primary beam on target
03:43 beam reached AIDA, fragmentation beam (see elog entry by DK for PID)
primary beam 300 Mev/u 40Ar, fragments in region of 34Si
started writing to file Dec19/R2 at rate ~ 4000 kb/s
histograms zeroed
03.49 system wide checks all ok
attachments 4-6:
temperatures ok
bias/leakage currents ok
good event statistics ok, except aida09 running slightly fast
04:03 attachments 7-8 - 1.8.H energy spectra
- no high energy events seen in aida10 -> checked ASIC control
- still no high energy events in aida10
04:09 attachment 9 - rate spectra for all FEE modules
04.16 attachment 10 - 1D hits for all FEE modules
04.19 beam stop - file R2_3
04:52 beam back on - file R2_6 - rate of ~ 5000 ions per spill
06:13 temperatures ok
good event statistics ok (aida09 still running fast) - attachment 11
detector bias/ leakage currents ok - see google spreadsheet
all system wide checks ok
1.8.H spectra - attachments 12/13
06:20 attachment 14 - rate spectra for all FEE modules
attachment 15 - 1D hits for all FEE modules
still no high energy events in aida10
|
|
Wed Apr 3 09:28:07 2019, CA, CB, TD, Wednesday April 3rd 2018 11x
|
Arrived at 09:54 - sequenced all FEE64 on
FEE64 not mounted properly - something happened to network overnight, see attachment 1,2 for AIDA system log
10.23 - FEE64 still not mounted properly, power cycling FEE64
10.30 - sequenced all FEE64 on
10.43 - FEE64 mounted successfully
- DAQ reset and setup - looks ok
- histogram enabled on aida12
- clock ReSYNC
- system wide checks - all FEE64 pass clock status check
- aida2 and aida12 fail ADC calibration
- SYNC pulse and SYNC error checks ok
- WR decoder check returns errors (attachment 3)
10.53 - FEE64 temperatures ok (attachment 4)
- detector bias/leakage current ok (attachment 5)
11.04 - TapeServer start - not writing to storage
- ASIC check load performed
- DAQ start
- ASIC control tab (attachment 6)
- Run control keeps reloading - has returned to 'reset' screen with aida11 showing "undefined" (attachment 7)
- good event statistics - all FEE64 showing 0 *except* aida09 (attachment 8)
11.20 - stopped tape server
- reset daq - returns error (attachment 9)
11.41 DAQ off, detector bias off, FEE64 off
13.30 detector bias on, FEE64 sequenced on
all browser tabs reopened
merger set back up - ok
mbs data relay set back up - ok
DAQ setup and started, initially ok
good event statistics (attachment 10)
16.25 FEE64 restarted, along with Merger, TapeServer, DAQ (not writing to file)
thresholds (slow comparator, fast comparator HEC, fast comparator LEC) raised to 0xff for testing.
aida10 stats appear reasonable, however wave events still reported despite being deactivated (see report entry
on DAQ stall) (attachment 11)
17.00 DAQ, TapeServer, Merger all stopped
Detector bias off
FEE64s switched off
coolant water still running (cannot be disabled due to lack of access)
AIDA in safe state |
|
Wed Apr 3 13:29:20 2019, CA, CB, TD, report - high DAQ stall
|
aida10 runs very fast for a period, then stops.
Have to restart Merger and TapeServer to get it running again.
As shown in attachment 1, aida10 records very high number of bad events.
Waveform for all FEE64s are disabled (see for example attachment 4) *BUT* note that 'wave events' are reported
in aida10 stats (attachment 1) |
|
Tue Nov 19 21:22:06 2019, CA, CB, DK, NH, 19/20 November 2019 - overnight shift 15x
|
19 November
Waiting for 40Ar beam to be delivered for tests.
Objective today is to assess degrader thickness suitable to stop 40Ar in 2nd DSSD layer.
Energy expected at 300 MeV/A, around 1E3-4 beam intensity - to be determined.
Tomorrow fragments could be expected.
DSSD / FEE64 pairing as follows
DSSD3 (most downstream): FEEs 9,10,11,12
DSSD2 (middle): FEEs 5,6,7,8
DSSD1 (most upstream): FEEs 1,2,3,4
+plastic scintillator furthest upstream
New AIDA Google spreadsheet created with leakage current, to monitor them during beamtime. First entry added. Bias attached.
https://docs.google.com/spreadsheets/d/12hgbrywB10hGsKt5uc2HnLfZymh8_uvM6cEASbbqnBE/edit#gid=813167023
DESPEC platform has already been moved. No effect on rates - see attached. FEE1 is missing one channel. FEE9, 10 noisy as expected. See attached.
System wide checks *all OK* except
Master clock failed (normal)
ADC calibration fails on 6 and 12. Recalibrated via FADC Align. All OK now.
MBS transfer OK. See attached.
Good event statistics OK. See attached.
FEE64 temperatures OK. See attached.
Merger and data rate OK. See attached.
Waiting for beam.
20 November
01:07 - Still no beam.
System wide checks OK.
Stats OK.
Temps OK.
MBS OK.
Merger and data rate OK.
02:35 - Beam is expected soon
Tried enabling writing data to disk. Merger crashed. FEE64 stopped responding to commands.
Reset Merger. FEE64 now respond to commands.
Tried enabling writing to disk again. Merger crashed once more. Reset merger.
Reset Tape Server interface. Now writing to disk.
No data currently being sent to MBS, but operators say they "are adding a new detector". May be normal.
DAQ & Merger going OK. Writing to Nov19/R1
No beam yet.
Temperatures OK
System wide checks OK
Stats OK. See attached.
MBS not working.
Merger and data rate OK. See attached.
~02:48 Beam on. 40Ar. 300 MeV/A, 500 pps. Degrader settings unclear.
03:54 Degrader settings have been changing between 4.5, 1 and 6 ug/cm2
Currently on 6 g/cm2.
AIDA seen through MBS shows no HE on detector 3, and Det 1 and 2 show no counts on positive y (FEE6 and 1). That is not the case from MIDAS.
04:30 The aim of the tests above was to have no 40Ar at all in DSSD3. It does not appear that 6 g/cm2 is sufficient, as we still see events. The reduction between 1 & 2 vs. 3 is about a factor of 2-10 depending on ADC channel. See attached.
~04:30 Moved to 6.2 g/cm2 degrader. See attached.
~04:45 Moved to 5.8 g/cm2. See attached.
~05:09 Moved to 6.4 g/cm2. See attached.
05:31 High energy channel spectra (1.8.H) for all FEE64 w/ 6.4 g/cm2 degrader - See attachments 13 & 14
05:38: Moved to 1.0 g/cm2. See attached. Difference is not huge which may indicate these spectra are misleading, as partly expected. A more detailed analysis will be carried out off line.
06:00: Beam stopped. ~R77. AIDA keeps running background over the day. Beam should restart around 11 pm later today. |
|
Wed Nov 20 20:36:14 2019, CA, CB, DK, NH, 20/21 November Night Shift 17x
|
21:35 (NH) - A few merger timestamp errors were appearing - Decided to powercycle all FEE64s, all checks OK.
No Screenshot due to forgetting before I closed merger
FEEs were often not stopping properly and needing merger reset as well.
Beam Plan:
40Ar 300 MeV/u Primary Beam
and/or
34Si Fragment from Ar
22:45
FEES were restart near these timestamps in the TapeData, evidently
-rw-rw-r--. 1 npg npg 2.0G Nov 20 21:17 R1_361
-rw-rw-r--. 1 npg npg 2.0G Nov 20 21:36 R1_362
Run number was kept the same (Tape Server was not stopped @ FEE restart?)
All system wide checks are okay (except of course the master clock fails)
Temps look okay (attachment 1, dated 21/Nov by mistake)
Leak currents all around 0.9 uA as before (attachment 2)
Stats are behaved. We don't have beam yet (attachment 3)
Merger and tape server happy (see attachment 4)
Data rate histos while beam is still off (see attachment 5)
==Nov 21==
0:00
TD noted that the data rates have gone up at some point (about double).
Now we increased the slow comparator thresholds from 0xa to 0xb (data rate drops from 24 Mb/s to 16 Mb/s).
We increased it again from 0xb to 0xc, but near that time the beam was also turned on, and the data rate was about 25 Mb/s.
Beam was turned on near R1_472 or a bit earlier.
TD requested waveforms, but at the moment they are not appearing. Will enable WFs after a condition change.
Beam seen in all three DSSSDs, see attachment 6. Beam was turned off before I could get more screencaps.
Degrader change ... near R1_481
00:32 waveforms now enabled under Run Control. They also appear to be enabled on ASIC1 undre LED and Waveform controls. However, we are unable to see the waveforms in the relevant histograms.
00:44 3 g/cm^2 inserted R1_488
00:45 3.2 g/cm^2 degrader in R1_489.
1:00 Several degraders and combinations are being tested since last note.
1:10 It seems 1 g/cm^2 of energy loss is not accounted for by LISE++. Trying to understand the reason.
1:28 Satisfied with a total degrader thickness of 2.6 g/cm^2 -- seems to calculate that fragments should implant mainly in DSSSD2? Issue seems to be resolved updating the LISE++ file.
Near R1_507
Can see the implant samples attached as #7 and #8
1:43 According to the stats, most of the beam energy is being deposited in DSSSD1. See #9.
Present beam burst repetition is around 0.1 Hz (each ~10 sec)
2:00 better example of stats histos (see #10)
2:45 tuning the FRS degraders (ones for RIB separation rather than AIDA implantation depth).
3:17 Just started fragments. Disregard previous filenames with "34Si". Finally no degrader in the FRS was needed to get Z separation.
R1_559
As the DAQs are not synchronized, a quick call to the BNC PB-5 pulser shifiting from 2 -> 50 -> 2 Hz was used to make a signal burst.
S4 degrader is 5.2 g/cm^2
3:22 Updated biases in Google Sheets.
Having trouble seeing the fragments in the AIDA histograms. Since we just started fragments, zero the histos.
3:35 Attachment 11, with 34Si fragments as Stats. Difficult to click Update on the Rates at the right time.
From the beam monitors, fragments should be at around 3000 cps.
3:45 (Up to now, some optics tuning with slits, etc)
3:46 Optics settings accepted, AIDA at R1_572, accept this condition for data runs on ~34Si fragments.
3:47 Sent a pulser burst 2 -> 50 -> 2 Hz
3:50 zero histograms again.
Temperatures attachment 12
Biases attachment 13
Statistics attachment 14
Merger / Tape Server attachment 15
3:56
StatHistos attachment 16
4:09 FRS starts new run #56, AIDA R1_583. Pulser 2-> 50 -> 2 Hz
4:56 Stat histograms in attachment 17. Not zeroed since noted at 3:50 |
|
Wed Mar 3 15:02:19 2021, CA, MBS command line startup
|
/MIDAS/Linux/startup/datarelaymbs |
|
Mon Mar 8 17:02:38 2021, CA, Monday 8th March 18:00 - 00:00 12x
|
18:00 ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 1
FEE Temperatures - attachment 2
Bias and leakage currents ok - attachment 3
18:24 Rate spectra - attachment 4
18:30 Merger ok - 4.4M data items/s
TapeServer ok - 43MB/s
Data forwarding to MBS ok
19:28 beam back after a brief outage - AIDA on file R9_736
FRS having DAQ issues
19:57 FRS DAQ/Scalars back
20:12 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 5
FEE Temperatures - attachment 6
Bias and leakage currents ok - attachment 7
Merger ok - 4.5M data items/s
TapeServer ok - 46MB/s
Data forwarding to MBS ok
20:33 No beam - AIDA on file R9_821
21:08 rates spectra - attachment 8
22:35 All system wide checks ok *except aida06 fails clock status 6*
Statistics - attachment 9
FEE Temperatures - attachment 10
Bias and leakage currents ok - attachment 11
Merger ok - 4.4M data items/s
TapeServer ok - 43MB/s
Data forwarding to MBS ok
temperatures slightly longer to load, and aida04 came up as not responding. Responded as normal after another reload.
23:07 media/ThirdDrive at 65% capacity
writing to file R9_1021
file compression currently up to R1_686
23:36 beam stable ~1e9 pps
rate spectra - attachment 12
|
|
Tue Mar 9 23:12:17 2021, CA, March 10th 00:00 - 08:00 20x
|
00:13 beam back, but some spills 'missing'
DESPEC keeping beam as is, but may stop later if it worsens
00:15 DAQ continues OK - file R20_622
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:20 System wide checks all OK *except*
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
00:24 still no bad timestamp errors in NewMerger since 16:30 UTC
00:30 FEE64 Temperatures OK - attachment 1
Good event statistics OK - attachment 2
Detector bias & leakage currents OK - attachment 3
Merger OK - 4.2M data items/s
TapeServer OK - 45 Mb/s
01:30 rate spectra - attachment 4
HEC spectra - attachment 5 & 6
note aida04 spectrum still not showing
02:07 beam off - file R20_760
02:11 beam back - file R20_764
02:44 System wide checks all OK *except*
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
still no bad timestamp errors in NewMerger since 16:30 UTC
FEE64 Temperatures OK - attachment 7
Good event statistics OK - attachment 8
Detector bias & leakage currents OK - attachment 9
Merger OK - 4.2M data items/s
TapeServer OK - 44 Mb/s
04:16 bad timestamp errors in NewMerger terminal, first since 16:30 UTC - attachment 10
System wide checks all OK *except*
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
FEE64 Temperatures OK - attachment 11
Good event statistics OK - attachment 12
Detector bias & leakage currents OK - attachment 13
05:03 Merger OK - 4.5M data items/s
TapeServer OK - 43 Mb/s
no bad timestamp errors for ~40 mins, data forwarding to MBS ok at usual rate
06:11 rates spectra - attachment 14
06:14 System wide checks
ADC Calibration
FEE64 module aida07 failed
FEE64 module aida10 failed
Calibration test result: Passed 10, Failed 2
Check WR decoder status
Base Current Difference
aida05 fault 0x1591 : 0x1593 : 2
aida06 fault 0xe23c : 0xe23e : 2
aida07 fault 0x743 : 0x745 : 2
aida08 fault 0xf84e : 0xf850 : 2
aida09 fault 0x5449 : 0x544a : 1
White Rabbit error counter test result: Passed 7, Failed 5
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Check FPGA Timestamp Errors
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
still no further bad timestamp errors in NewMerger terminal
collected all WR and FPGA errors from baseline, system wide checks ok
FEE64 Temperatures OK - attachment 15
Good event statistics OK - attachment 16
Detector bias & leakage currents OK - attachment 17
06:24 AIDA writing to file R20_1080
Merger OK - 4.5M data items/s
TapeServer OK - 46 Mb/s
06:26 another burst of bad timestamp errors - attachment 18
performed system wide checks again - all ok aside from aida07/aida10 calibration
07:32 HEC spectra - attachments 19/20
07:35 no beam
|
|
Thu Mar 11 23:07:23 2021, CA, March 12th 00:00 - 08:00 shift 15x
|
00:10 DAQ continues OK - file R30_372
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:17 System wide checks all OK
FEE64 Temperatures ok - attachment 1
Good event statistics ok - attachment 2
detector bias and leakage currents ok - attachment 3
00:20 DESPEC on run 145
00:24 Merger ok ~45M items/s
Tapeserver ok ~15MB/s
01:00 aida07 crashed
recovered, but all FEE64 crash shortly after
called OH - stop DAQ, TapeService, Merger
01:20 all FEE64 powercycled, AIDA restart
01:30 AIDA recovered, DAQ now running - writing to R33
system wide checks ok *except*
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
01:52 FEE64 Temperatures ok - attachment 4
-had to reload a few times to work, but otherwise ok
Good event statistics - attachment 5
-aida05 and aida08 running faster than before
detector bias and leakage currents ok - attachment 6
02:04 writing to file R33_34
Data forwarding to MBS ok
AIDA ASIC settings ok
02:11 beam off
02:13 attempted to recalibrate aida07 and aida09 in FADC Align and Control - calibration still fails
02:32 beam back - writing to file R33_48
04:30 DESPEC having issues with Go4 crashing
ucesb reports AIDA/FATIMA/bplast timewarp events
System wide checks:
Clock error:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Calibration:
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
White Rabbit:
Base Current Difference
aida07 fault 0xcdd1 : 0xcdd2 : 1
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
FEE64 Temperatures ok - attachment 7
Good event statistics ok - attachment 8
detector bias and leakage currents ok - attachment 9
04:52 analyzer output for R33_113 - attachment 10
no timewarps
aida06 dead time? (ignore idle time and rates)
05:02 Merger 5M data items/s
TapeServer 17 MB/s
05:15 DESPEC believe issue with bplast TAMEX causing ucesb issues/Go4 crash
They disable bplast and FATIMA TAMEX histograms in their online analysis - ucesb/go4 much more stable now
05:25 rates spectra - attachment 11
05:29 error message in MBS relay terminal - otherwise data forwarding to MBS ok - attachment 12
06:29 system wide checks:
Clock error:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Calibration:
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
White Rabbit:
Base Current Difference
aida05 fault 0x7a56 : 0x7a58 : 2
aida06 fault 0x65bb : 0x65bd : 2
aida07 fault 0xcdd1 : 0xcdd4 : 3
aida08 fault 0x2ab3 : 0x2ab5 : 2
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
rest ok
06:32 FEE64 Temperatures ok - attachment 13
Good event statistics ok - attachment 14
detector bias and leakage currents ok - attachment 15
06:41 Merger 5.1M data items/s
TapeServer 17 MB/s
Data forwarding to MBS ok
08:54 restart MBS relay, requested by NH
|
|
Sat Mar 13 23:08:08 2021, CA, 14th March 00:00-08:00 22x
|
00:08 DAQ continues ok - writing to file R46_221
ASIC settings 2019Dec19-16.19.51
DSSSD#1 slow comparator 0xa
DSSSD#2 slow comparator 0xa
DSSSD#3 slow comparator 0xd
BNC PB-5 Pulser
Amplitude1.0V
Attenuation x1
Frequency 2Hz
tau_d 1ms
- polarity
Delay 250ns, tail pulse
00:09 system wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida09 failed
Calibration test result: Passed 11, Failed 1
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x82a0 : 0x82a4 : 4
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
Base Current Difference
aida07 fault 0x2 : 0x3 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 11 6 2 5 1 3 2 2 3 4 10 : 54492
aida02 : 12 7 3 4 3 3 3 2 3 3 6 : 36440
aida03 : 4 4 2 4 2 3 2 2 3 3 6 : 36048
aida04 : 18 9 4 2 1 3 2 4 1 4 15 : 73936
aida05 : 31 11 8 7 6 4 2 18 7 4 7 : 55220
aida06 : 36 8 18 4 4 2 1 4 3 2 5 : 31088
aida07 : 26 7 4 3 1 2 1 3 3 3 6 : 36224
aida08 : 25 12 5 4 4 3 2 3 3 3 6 : 36884
aida09 : 2 9 4 2 2 2 2 3 3 3 6 : 36432
aida10 : 11 3 4 6 4 3 1 3 3 3 6 : 36548
aida11 : 21 8 5 0 1 3 3 3 2 3 6 : 35748
aida12 : 10 6 6 2 4 4 1 3 3 3 6 : 36600
00:14 FEE64 Temperatures ok - attachment 1
Good event statistics ok - attachment 2
Detector bias/leakage currents ok - attachment 3
00:17 Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
01:30 beam off
01:31 beam back
02:19 system wide checks
same as entry at 00:09, *except*
Base Current Difference
aida07 fault 0x82a0 : 0x82a5 : 5
White Rabbit error counter test result: Passed 11, Failed 1
02:22 FEE64 Temperatures ok - attachment 4
Good event statistics ok - attachment 5
Detector bias/leakage currents ok - attachment 6
02:30 Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
no further bad timestamp errors in NewMerger terminal
02:40 ... burst of bad timestamp errors in NewMerger terminal - attachment 7
02:50 ... and sure enough lose connection to aida07 after checking stats
DAQ stop, Merger/TapeServer closed
telnet aida07 -> sent reboot command
mounted ok, reset DAQ, TapeService and Merge
restarted DATA forwarding to MBS
all looks to be ok now -> writing to R56
03:09 system wide checks:
FEE64 module aida09 global clocks failed, 6
Clock status test result: Passed 11, Failed 1
Understand status as follows
Status bit 3 : firmware PLL that creates clocks from external clock not locked
Status bit 2 : always logic '1'
Status bit 1 : LMK3200(2) PLL and clock distribution chip not locked to external clock
Status bit 0 : LMK3200(1) PLL and clock distribution chip not locked to external clock
If all these bits are not set then the operation of the firmware is unreliable
FEE64 module aida07 failed
FEE64 module aida09 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
Base Current Difference
aida07 fault 0x2 : 0x5 : 3
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 13 5 3 7 2 2 2 2 3 4 10 : 54508
aida02 : 2 7 3 2 2 3 1 4 3 3 6 : 36784
aida03 : 8 8 1 3 2 3 1 3 3 3 6 : 36304
aida04 : 20 9 2 3 0 3 2 4 1 4 15 : 73880
aida05 : 51 74 70 54 53 49 22 18 7 4 7 : 72188
aida06 : 15 15 3 9 1 2 2 4 3 2 5 : 31044
aida07 : 24 6 2 3 3 4 2 3 2 3 7 : 39888
aida08 : 19 15 1 2 1 3 1 4 3 3 6 : 36820
aida09 : 18 4 2 7 3 4 1 3 3 3 6 : 36648
aida10 : 20 7 1 1 1 3 3 2 3 3 6 : 36216
aida11 : 25 7 5 3 4 2 1 4 2 3 6 : 35916
aida12 : 3 3 2 2 3 2 3 2 3 3 6 : 36164
03:18 FEE64 Temperatures ok - attachment 8
Good event statistics ok - attachment 9
Detector bias/leakage currents ok - attachment 10
Merger ok - 4.4M items/s
TapeServer ok - 14 MB/s
03:29 more bad timestamp error messages - attachment 11
03:52 analyser output on R56_35 - attachment 12
no timewarps
03:53 DAQ continues ok, writing to file R56_41
05:18 system wide checks:
same as entry at 03:08, *but* with additional WR decoder status error
Base Current Difference
aida07 fault 0x8c74 : 0x8c77 : 3
White Rabbit error counter test result: Passed 11, Failed 1
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
05:19 FEE64 Temperatures ok - attachment 14
Good event statistics ok - attachment 13
Detector bias/leakage currents ok - attachment 15
Merger ok - 4.4M items/s
TapeServer ok - 15 MB/s
05:25 rate spectra - attachment 16
05:33 AIDA implant XY hit patterns - attachments 17/18/19
05:56 BuTIS interface control set to 0x3 at 04.52 UTC
07:06 system wide checks - same error messages as entry at 05:18
07:08 FEE64 Temperatures ok - attachment 20
good event statistics ok - attachment 21
detector bias/leakage currents ok - attachment 22
Merger ok - 4.3M items/s
TapeServer ok - 15 MB/s
07:22 no bad timestamp errors since 02:30 UTC |
|
Tue Apr 20 07:06:20 2021, CA, April 20th 08:00 - 12:00
|
08:09 DAQ crashes at start of shift (!)
again happens at same time as beam being lost
All FEE64 lost connection
Beam downtime for next 3-4 hours
Performed powercycle and reset
08:40 AIDA back up and running, writing to R36
08:42 *all* system wide checks ok
temperatures ok - attachment 1
statistics ok - attachment 2
detector bias/leakage currents ok - attachment 3
08:57 options file size check;
Collecting the file size of each FEE64 Options CONTENTS file to check they are all the same
FEE : aida01 => Options file size is 1025 Last changed Tue Apr 20 08:22:14 CEST 2021
FEE : aida02 => Options file size is 1014 Last changed Fri Apr 16 00:56:20 CEST 2021
FEE : aida03 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida04 => Options file size is 1025 Last changed Sat Apr 17 06:07:36 CEST 2021
FEE : aida05 => Options file size is 1025 Last changed Fri Apr 16 00:53:25 CEST 2021
FEE : aida06 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida07 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida08 => Options file size is 1014 Last changed Wed Apr 14 21:52:04 CEST 2021
FEE : aida09 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida10 => Options file size is 1014 Last changed Wed Apr 14 21:52:06 CEST 2021
FEE : aida11 => Options file size is 1014 Last changed Wed Apr 14 21:52:05 CEST 2021
FEE : aida12 => Options file size is 1025 Last changed Mon Apr 19 09:05:25 CEST 2021
08:59 no error messages in database check terminal
stats ok
09:16 waveforms - attachments 4 & 5
09:30 no error messages in database check terminal
stats ok
10:07 no error messages in database check terminal
stats ok
10:30 aida01, aida02, aida03, aida04, aida07 lost connection, DAQ crashes
powercycle attempted, but FEE modules remain unconnected
TD performs further power cycle
11.12 reboot aida05
DAQ reset/setup
system wide checks
sync asic clocks OK
clock status & fpga timestamp errors OK
WR decoder status
Base Current Difference
aida09 fault 0xb307 : 0xb307 : 0
aida09 : WR status 0x40
aida10 fault 0x3d6c : 0x3d6c : 0
aida10 : WR status 0x40
aida11 fault 0xa610 : 0xa610 : 0
aida11 : WR status 0x40
aida12 fault 0x7c7 : 0x7c7 : 0
aida12 : WR status 0x40
White Rabbit error counter test result: Passed 8, Failed 4
Understand the status reports as follows:-
Status bit 3 : White Rabbit decoder detected an error in the received data
Status bit 2 : Firmware registered WR error, no reload of Timestamp
Status bit 0 : White Rabbit decoder reports uncertain of Timestamp information from WR
NH had access to area, unplugged and replugged HDMI cables from aida09, aida10, aida11, aida12 root to MACB (at both ends)
earlier on, Akash secured HDMI cable to aida07
11:55 AIDA powercycle and reset, DAQ ok now
writing to file NULL/R45
12:00 CB takes over |
|
Tue Apr 20 23:08:22 2021, CA, Wed Apr 21 00:00-08:00 23x
|
00:00 - CA takes over
00:15 - stats/database/ucesb ok - attachment 1
- another burst of bad timestamp error messages in NewMerger terminal - but all FEE64 still ok - attachment 2
- Merger/TapeService ok
00:21 - beam still off while UNILAC having problems
00:26 - beam is back - rate spectra - attachment 3
00:32 - ucesb following beam return - attachment 4
- implant rates reaching highs of ~1kHz
00:35 - Grafana leakage current monitor last 6 hours - attachment 5
01:00 *all* system wide checks ok
stats ok - attachment 6
temperatures ok - attachment 7
detector bias / leakage currents ok - attachment 8
db check ok / ucesb ok
still observe periodic bursts of bad timestamp errors in NewMerger terminal - FEE64s still linked with merger ok
01:30 AIDA crashes just as I make check, most FEE64 lost connection -> powercycle and restart MIDAS
01:53 AIDA reset completed - writing to file NULL/R48
01:56 all system wide checks ok *except*
FPGA error counter
Base Current Difference
aida12 fault 0x0 : 0x1 : 1
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
ADC calibration
FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
02:01 good event statistics ok - attachment 9
temperatures ok - attachment 10
detector bias / leakage currents ok - attachment 11
db check / ucesb ok
02:20 beam stop
02:21 beam back
02:30 stats/ucesb/db checks all ok
03:00 stats/ucesb/db checks all ok - attachment 12
03:30 stats/ucesb/db checks all ok
04:00 system wide checks ok, except;
ADC calibration
FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
FPGA errors
Base Current Difference
aida12 fault 0x0 : 0x9 : 9
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
04:03 beam off briefly
04:05 good event statistics ok - attachment 13
04:07 beam back
temperatures ok - attachment 14
bias / leakage currents ok - attachment 15
db check / ucesb ok - attachment 16
implant rates in AIDA peaking at ~1.5kHz
04:30 stats/ucesb/db checks all ok
04:48 no beam
04:58 beam back
05:00 stats/ucesb/db checks all ok - attachment 17
05:30 stats/ucesb/db checks all ok
06:00 system wide checks ok, except;
ADC calibration
FEE64 module aida04 failed
FEE64 module aida05 failed
Calibration test result: Passed 10, Failed 2
If any modules fail calibration , check the clock status and open the FADC Align and Control browser page to rerun calibration for that module
FPGA errors
Base Current Difference
aida12 fault 0x0 : 0x9 : 9
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
FEE64 Linux Memory Info
Returned 0 0 0 0 0 0 0 0 0 0 0 0
Mem(KB) : 4 8 16 32 64 128 256 512 1k 2k 4k
aida01 : 5 5 3 3 2 3 2 3 2 3 7 : 39628
aida02 : 22 11 3 0 2 3 2 3 2 3 7 : 39648
aida03 : 20 10 1 1 2 2 1 4 2 3 7 : 39760
aida04 : 17 10 2 2 3 3 2 3 2 3 7 : 39732
aida05 : 1 8 3 2 3 2 2 3 3 3 7 : 40564
aida06 : 5 6 2 3 2 2 3 3 2 3 7 : 39748
aida07 : 18 4 7 0 1 3 1 4 2 3 7 : 39832
aida08 : 22 10 3 3 2 2 3 3 2 3 7 : 39864
aida09 : 3 6 2 2 0 3 1 4 2 3 7 : 39708
aida10 : 2 4 3 2 1 2 2 3 2 3 7 : 39384
aida11 : 20 8 3 1 1 3 3 2 2 3 7 : 39328
aida12 : 16 6 3 1 0 3 2 3 2 3 7 : 39488
06:02 good event statistics ok - attachment 18
FEE64 temperatures ok - attachment 19
detector bias / leakage currents ok - attachment 20
06:06 db check and ucesb ok - attachment 21
NewMerger terminal hasn't flagged up any more errors since ~ 03:00
06:30 stats/ucesb/db checks all ok
DESPEC restarts timesorter
07:00 stats/ucesb/db checks all ok
07:30 stats/ucesb/db checks all ok - attachment 22
grafana leakage currents last 1 hour - attachment 23
|
|
Wed Apr 21 23:05:39 2021, CA, 22nd April 00:00 - 08:00 16x
|
00:07 stats/ucesb/DB checks all ok
00:30 all system wide checks ok *except* fpga errors
Base Current Difference
aida12 fault 0x0 : 0x1f : 31
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
statistics ok - attachment 1
temperatures ok - attachment 2
detector bias / leakage currents ok - attachment 3
no error messages in DB terminal & ucesb ok - attachment 4
01:00 stats/ucesb/DB checks all ok
01:30 stats/ucesb/DB checks all ok
01:53 issues with beam dropping in and out - ESR working to stabilise this
02:00 stats/ucesb/DB checks all ok
02:09 beam back in a stable state
02:30 all system wide checks ok *except* fpga errors
Base Current Difference
aida12 fault 0x0 : 0x29 : 41
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
02:33 statistics ok - attachment 5
temperatures ok - attachment 6
detector bias / leakage currents ok - attachment 7
no error messages in DB terminal & ucesb ok - attachment 8
02:45 beam off - potentially serious issue, awaiting update from FRS
03:00 significant issues with UNILAC, beam not expected back for a while
03:05 stats/ucesb/DB checks all ok
03:32 stats/ucesb/DB checks all ok
03:55 beam is back
04:00 stats/ucesb/DB checks all ok
04:30 all system wide checks ok *except* fpga errors
Base Current Difference
aida12 fault 0x0 : 0x2b : 43
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
04:32 statistics ok - attachment 9
temperatures ok - attachment 10
detector bias / leakage currents ok - attachment 11
no error messages in DB terminal & ucesb ok - attachment 12
05:00 stats/ucesb/DB checks all ok
05:30 stats/ucesb/DB checks all ok
06:00 stats/ucesb/DB checks all ok
06:30 all system wide checks ok *except* fpga errors
Base Current Difference
aida12 fault 0x0 : 0x34 : 52
FPGA Timestamp error counter test result: Passed 11, Failed 1
If any of these counts are reported as in error
The ASIC readout system has detected a timeslip.
That is the timestamp read from the time FIFO is not younger than the last
06:32 temperatures ok - attachment 13
statistics ok - attachment 14
detector bias / leakage currents ok - attachment 15
no error messages in DB terminal & ucesb ok - attachment 16
07:00 stats/ucesb/DB checks all ok
07:30 stats/ucesb/DB checks all ok
07:59 stats/ucesb/DB checks all ok
|
|